360 cameras such as Insta360 One RS and Qoocam 8K can now generate impressive 3D models quickly with the help of AI. Here is a tutorial and samples.
360 cameras have been increasingly popular for third person view videos as well as for virtual tours. Now 360 videos can also be converted into 3D models using a technology called NeRF.
NeRF (Neural Radiance Fields) is a machine learning model for 3D object reconstruction. It is able to infer the 3D structure and texture of objects from 2D images, by using a neural network that learns to predict the 3D structure of the object from multiple 2D views of it. NeRF can be used for applications such as 3D object reconstruction from images, 3D scene understanding, and augmented reality.
In simple and layman terms, if you have a 360 camera you can now create a Matterport style 3D model using AI.
Here is an example of what is possible with a simple 360 video
So how does one do this ?
To create a 3D model using NeRF, a photographer first needs to capture a 360-degree video using a 360 camera. These cameras are designed to capture footage from every angle, allowing the photographer to capture the entire scene in a single shot. Once the video has been captured, the photographer can then use NeRF to convert it into a 3D model. This is done by feeding the video into a neural network, which is trained to analyze the video and create a 3D model based on the information it contains.
The resulting 3D model can then be used in a variety of ways, such as for virtual reality experiences, 3D printing, or simply for viewing on a computer.
If you are a programmer and you think you can give this a shot, follow this 40 min tutorial that will help you.
One of the key benefits of using NeRF to convert 360-degree videos into 3D models is that it allows for a high level of detail and accuracy. Because the process uses machine learning, the resulting 3D models are able to capture the fine details and nuances of the scene, resulting in highly realistic and lifelike models.
In addition, using NeRF to create 3D models from 360-degree videos is also a relatively fast and efficient process. Because the neural network is trained to analyze the video and create the 3D model, the process can be completed quickly, allowing photographers to quickly and easily create 3D models from their 360-degree footage.
Overall, the use of NeRF to convert 360-degree videos into 3D models is a powerful tool for photographers and videographers. By combining the immersive viewing experience of 360-degree videos with the detail and accuracy of 3D modeling, photographers are able to create unique and engaging content that can be used in a variety of applications.
Here are other samples of 3D models generated from 360 videos with NeRF by Tomonori Taniguchi.
Here is a 360 video of a 3D model shot on the Insta360 One RS 1-inch 360.
Another sample of a 3D model from the 1-inch 360:
This one was shot with Qoocam 8K:
Interesting! Now to see if somebody can work this into a simple installable program. I’ve watched the howto video a bit but I was lost pretty fast lol…
There are Iphone apps that allow you to capture and create these 3D spaces, not large spaces but you can try them for free. The app that I use is Luma on the iOS.
Hi, i think i installed 2 nerf programs but only managed to get 1 started.
i saw the demo files but never managed to get it running. apparently it need an huge amount of GPU to build a simple scene so i’d like to know what specs were used since my poor 6gb gpu was not enough for 60 pics under 1 megapixel (or something this insignificant)
Yes, the processing needed for this is not trivial especially for anything that’s NOT like a 10 sec clip.
I think the specs needed are in the youtube video
Amazing feeling like it is so real, tree makes something how it looks like
Excellent article!!! Is there a tutorial on how to do this? Thanks a Lot!
best to look at the article referenced, personally I have not tried it yet.
Interesting, isn’t it?
I’ve been following the topic for a while but for now neither my current GPU nor my free time allows me to carry it out. I am so looking forward to the simplification of the process and system requirements to be able to create such walkthroughs using NERFs.
As an interior design photographer I already have a list of possible clients that might be interested.
Thanks for posting. Keep us updated!