What if you could create a 3D model of a space, simply by walking around and recording a 360 video of it with an off-the-shelf consumer 360 camera? Maxime Lhuillier from CNRS/Institut Pascal/UCA demonstrated just such a technology, and now you can check out the amazing demo video and explore two 3D models!
Photogrammetry alternative
Nowadays, it is possible to create a 3D model of a space by taking hundreds of photographs of a space and using computers to analyze those photos through photogrammetry and computer vision methods. This is a laborious process, and is especially time-consuming and resource-intensive for large scenes. However, Institut Pascal has developed software that can convert a 360 video taken by a consumer 360 camera into a 3D model.
How it works
For this demo, Maxime Lhuillier attached a Garmin Virb 360 onto a helmet and walked about a space while recording a 360 video. Using only that video, their software has been able to generate a 3D model of a space as complex as a forest. The following video shows walkthroughs in a 3D model of a forest reconstructed from videos taken by a Virb 360:
Compared to conventional photogrammetry, this method poses several additional technical hurdles: not only are there large scenes, but with compressed videos, the image quality is lower than those of still images or professional cameras. In addition, backlit portions of the videos have less visible detail after image compression, making 3D reconstruction even more difficult.
Remarkably, the 360 video was converted into a 3D model using only a laptop. There are several limitations to this technology: the scene must be static, there must be sufficient texture and light in the scene, the camera motion must be slow enough, and the walking or biking must be slow. The goal of this technology is not accuracy for metrology applications but to approximate a complex scene for visualization using consumer cameras. It’s exciting to imagine the many potential commercial and industrial applications of this technology, such as for real estate, mapping, or documenting a construction site.
Explore these demos
Here are two 3D models that you can explore in Sketchfab. These two models were generated from videos taken by a 360 camera rig consisting of four helmet-mounted Gopro Hero 3 cameras recording video at 100 fps with frame-accurate synchronization. You can explore these models in VR with an HTC Vive (reviewed here) or Oculus Rift (reviewed here) or even just Google Cardboard using a browser that supports WebVR, such as Firefox Nightly.
First, here is a 2.5km long street in a city (only a few regions are textured due to texture storage limitation). This model was generated from biking for 2.5 km, and the gray squares show 2,615 keyframes:
Second, here is a 3D model of a forest. This one was recorded from a video while walking around a forest for 4 minutes. The gray squares show 878 keyframes:
Both the software and method are currently under development, and researchers are working on improvements for all steps of the method (matching, structure from motion, self-calibration, loop closure, surface reconstruction, and texturing). The software is not available for demonstration, nor does Institut Pascal have plans for selling the software. However, there are other companies developing software for similar purposes. Meanwhile, what do you think of the technology and these demos? Let me know in the comments!
Thank you very much to Maxime Lhuillier for sharing this demo! Maxime is a CNRS researcher at the Institut Pascal, a laboratory of Université Clermont Auvergne in Aubière, France. You can check out his account at YouTube and Sketchfab to see other 3D models generated using this technology.
This is all fantastic stuff.
I’m hoping that in the long term, we have some AI algorithms that can “auto-fix” the 3D defects in real time. (Artificial Intelligence: “…That looks like tree bark, let me fix that texture…. Now, that looks like a leaf, let me fix that texture…. Now that looks like a road; let me flatten it out to a normal looking road…”). Then maybe we can create true Holodecks out of 360 videos.
(Chief Blur Buster here, I write about VR announcements as well)
Thanks Chief Blur Buster! Nice to meet you!
Best regards,
Mic
Insert a drone, let the drone make the scan, then import in a 3D environment in real time. People can analyze the site for forensic data even, plan tacticals, etc.
Hi Khannea. Just to clarify, Cupix does not convert the 360 photos into 3D in realtime. The process takes several hours.
Best regards,
Mic
Whenever I see “jaw dropping” in a story title, credibility takes a dive. This is a cheap click-bait term. Looking at the generated 3D models, they are quite rough. not that impressive.
Thanks for the feedback Bill. The 3d models are indeed rough but what is amazing to me is that the software generated these models from a 360 video.
Best regards,
Mic
This is Great stuff (Y)
I think so too Priyadarshan!
Best regards,
Mic
This looks great, but the depth in the 3D model generated is not up to the mark. Thanks for sharing though.
Hi Venkatesh. Yes the 3D model is quite simple. I’m confident that as the technology improves we will see more detailed models. Best regards, Mic
Great job, guys. From Russia with Love!
Hello and welcome to 360 Rumors! Good to have you here!
Best regards,
Mic
This all remember me to the free Opensource Software Blender make a mix in VR with 360 Videos and Animations. Great work
https://youtu.be/mN0zPOpADL4
wow !!!!!! this is such a huge breakthrough!!! looking forward for a commercial license.
Hi Luis! Institut Pascal said they don’t plan to commercialize it. However, you might be interested in something similar AND will be available, which is this: https://360rumors.com/2017/12/true-vr-camera-everyone-giroptic-io-will-6dof-vr-capture-capability.html
This is very interesting. I do hope software such as this will become available.
Hi Alison. Stay tuned for my upcoming post on Giroptic’s software for doing this with a $200 camera.
Best regards,
Mic
I find it interesting that you can now create 3d models from such videos. But the quality of the resulting models still has to be greatly improved in order for them to have a practical benefit.
It’s a nice turnkey solution, though far from professional results. It still really comes down to exporting an image sequence and depth maps sequence (if possible), then using Photoscan or RealityCapture to create your point cloud or mesh model. Anything less is just unuable, albeit novel.
Great work! This is exactly the kind of thing I was looking for (as-built space documentation, reduce trips to the site, without the $40k price tag). What type of format are the 3d models generated into? Sketchfab is a new software to me, is it like sketchup but with different capabilities? I hope to hear more about this soon.
i don’t remember the format, but it is a standard 3d model format. sketchfab is more for showing than creating 3d models.
This demo already surpasses Google’s Street View photogrammetry quality. If only the technology could be democratized so more people could add to the global point cloud database.
i agree that would be ideal! i think that’s what google is trying to do, which is why it is accepting even 4K photos.
Is this processing software available for download to play with? I’ve got a GoPro Fusion and Insta360 ONE X.
nope