Facebook will soon enable you to step inside your photos and videos. I was lucky enough to see a demo of this technology at IVRPA 2018. Guess what? It really works.
With this technology, Facebook can analyze the photos and videos you’ve taken and use computer vision to reconstruct a 3D model of the place where the photos and videos were taken. You can see it discussed starting at around 1 hr 47 mins of this video of the F8 keynote day 1:
I got to try it out for myself in May, at IVRPA 2018 in Tokyo where I met some representatives from Facebook’s Immersive Media group. Here’s a video of me trying it out (the video is from a segment in my Samsung Gear 360 review video — please watch the whole video):
Although the video wasn’t super detailed, the critical thing was that the proportions of the objects in the 3D model were very realistic and 1:1 scale. I could really get a sense of the place. As I moved around, I could see Photos suspended in the air. When I click on them, the photos would be enlarged so I could see them more clearly.
Their 3D reconstruction software was smart enough to recognize which objects did not belong in the room and remove them. For example, if you took photos with people, the software could recognize that the people are not a part of the room, so the 3D model won’t include the people.
Crucially for us, the Facebook reps said that 360 videos are the ideal medium from which to generate a 3D model. Facebook will be able to create a 3D model from 360 videos much more easily than regular photos or videos. With only a few seconds of a 360 video, you’ll be able to get a 3D model (compare to this Institut Pascal videogrammetry demo). This means 360 cameras will become even more indispensable for preserving memories.