360 Camera News and Info VR News and Info

Jaw-dropping software converts 360 video into 3D model for VR

Institut Pascal's software converts 360 video into 3D model
Institut Pascal’s software converts 360 video into 3D model

What if you could create a 3D model of a space, simply by walking around and recording a 360 video of it with an off-the-shelf consumer 360 camera?  Maxime Lhuillier from CNRS/Institut Pascal/UCA demonstrated just such a technology, and now you can check out the amazing demo video and explore two 3D models!

Photogrammetry alternative

Nowadays, it is possible to create a 3D model of a space by taking hundreds of photographs of a space and using computers to analyze those photos through photogrammetry and computer vision methods.  This is a laborious process, and is especially time-consuming and resource-intensive for large scenes.  However, Institut Pascal has developed software that can convert a 360 video taken by a consumer 360 camera into a 3D model.

How it works

For this demo, Maxime Lhuillier attached a Garmin Virb 360 onto a helmet and walked about a space while recording a 360 video.  Using only that video, their software has been able to generate a 3D model of a space as complex as a forest. The following video shows walkthroughs in a 3D model of a forest reconstructed from videos taken by a Virb 360:

Compared to conventional photogrammetry, this method poses several additional technical hurdles: not only are there large scenes, but with compressed videos, the image quality is lower than those of still images or professional cameras.  In addition, backlit portions of the videos have less visible detail after image compression, making 3D reconstruction even more difficult.

Remarkably, the 360 video was converted into a 3D model using only a laptop.   There are several limitations to this technology: the scene must be static, there must be sufficient texture and light in the scene, the camera motion must be slow enough, and the walking or biking must be slow.  The goal of this technology is not accuracy for metrology applications but to approximate a complex scene for visualization using consumer cameras.  It’s exciting to imagine the many potential commercial and industrial applications of this technology, such as for real estate, mapping, or documenting a construction site.

Explore these demos

Here are two 3D models that you can explore in Sketchfab.  These two models were generated from videos taken by a 360 camera rig consisting of four helmet-mounted Gopro Hero 3 cameras recording video at 100 fps with frame-accurate synchronization.  You can explore these models in VR with an HTC Vive (reviewed here) or Oculus Rift (reviewed here) or even just Google Cardboard using a browser that supports WebVR, such as Firefox Nightly.

First, here is a 2.5km long street in a city (only a few regions are textured due to texture storage limitation).  This model was generated from biking for 2.5 km, and the gray squares show 2,615 keyframes:

Second, here is a 3D model of a forest.  This one was recorded from a video while walking around a forest for 4 minutes.  The gray squares show 878 keyframes:

Both the software and method are currently under development, and researchers are working on improvements for all steps of the method (matching, structure from motion, self-calibration, loop closure, surface reconstruction, and texturing).   The software is not available for demonstration, nor does Institut Pascal have plans for selling the software.   However, there are other companies developing software for similar purposes.  Meanwhile, what do you think of the technology and these demos? Let me know in the comments!

Thank you very much to Maxime Lhuillier for sharing this demo!  Maxime is a CNRS researcher at the Institut Pascal, a laboratory of Université Clermont Auvergne in Aubière, France.  You can check out his account at YouTube and Sketchfab to see other 3D models generated using this technology.

About the author

Mic Ty

1 Comment

Click here to post a comment

  • This is all fantastic stuff.

    I’m hoping that in the long term, we have some AI algorithms that can “auto-fix” the 3D defects in real time. (Artificial Intelligence: “…That looks like tree bark, let me fix that texture…. Now, that looks like a leaf, let me fix that texture…. Now that looks like a road; let me flatten it out to a normal looking road…”). Then maybe we can create true Holodecks out of 360 videos.

    (Chief Blur Buster here, I write about VR announcements as well)