Capturing 3D digital twins with photogrammetry and NeRF

 

Important note before you start reading !

This article is a bit of an experiment as it was mostly written by ChatGPT, a new generative Artificial Intelligence (AI) tool introduced by OpenAI in November 2022. All text marked in blue below was not written by a human being!

What is photogrammetry?

Photogrammetry is a technique used to measure and interpret the properties of an object from photographs. It involves using photographs of an object or scene taken from different viewpoints to create a 3D model or map of the object or scene.

Photogrammetry has a wide range of applications, including surveying and mapping, 3D modeling, and even archaeological research. It can be used to create accurate and detailed 3D models of objects, buildings, landscapes, and other physical features.

To perform photogrammetry, specialized software is used to analyze the photographs and extract information about the size, shape, and position of the objects in the scene. This information is then used to create a 3D model or map of the object or scene.

Photogrammetry is a powerful tool that allows us to measure and interpret the properties of physical objects and environments in a precise and accurate manner.

Mozart statue captured with photogrammetry

In this particular case, we created a digital twin of the Mozart statue, which is quite tall and physically located on the Mozart square in the city center of Salzburg. Due to tight restrictions in this high traffic pedestrian area, we were not allowed to use a drone to take the required pictures. Instead, we had to get creative with a long pole to reach the photo angles we needed. Here is the end result of the 3D digital twin.

Place Mozart into your own space with augmented reality (AR)

What are Neural Radiance Fields (NeRF) ?

Neural radiance fields (NRFs) are a type of machine learning model that can be used to synthesize images of 3D objects or scenes. NRFs are trained on large datasets of 3D objects or scenes, and they use this training data to learn the relationships between the 3D geometry of an object or scene and the 2D images that are produced when that object or scene is viewed from different angles.

Once trained, an NRF can be used to generate high-quality images of a 3D object or scene from a single 2D image or from a set of 2D images taken from different viewpoints. NRFs can also be used to generate novel views of an object or scene that were not present in the training data.

NRFs are a relatively new technology and are still being actively researched and developed. They have the potential to revolutionize the way 3D objects and scenes are synthesized and rendered, and they may have applications in fields such as computer graphics, augmented and virtual reality, and robotics.

Another statue captured with NeRF

What are the advantages of neural radiance fields over photogrammetry?

There are a few potential advantages of using neural radiance fields (NRFs) over photogrammetry for synthesizing images of 3D objects or scenes:

1. Speed: NRFs can generate high-quality images of 3D objects or scenes very quickly, making them well suited for applications that require real-time rendering or image synthesis.
2. Accuracy: NRFs can produce highly realistic images of 3D objects or scenes, as they are trained on large datasets of 3D objects or scenes and learn to accurately model the relationships between 3D geometry and 2D images.
3. Flexibility: NRFs can be trained to synthesize images of a wide variety of 3D objects or scenes, making them more flexible than photogrammetry, which is limited to measuring and interpreting the properties of physical objects.
4. Ease of use: NRFs can be trained and used without requiring specialized equipment or expertise, making them more accessible than photogrammetry, which typically requires specialized software and training.

It’s worth noting that NRFs are a relatively new technology and are not yet as widely used or well understood as photogrammetry. In addition, NRFs may not be suitable for all applications that involve synthesizing images of 3D objects or scenes, and photogrammetry may still be the preferred approach in some cases.