As I posted previously, a spectator mode will be added to other games on the HTC Vive (reviewed here). Here is how you can use the spectator mode to make a mixed reality video.
INTRODUCTION
A mixed reality video is one of the most effective ways of showing what it’s like to be in VR. Here is a sample.
OVERVIEW
Recording a mixed reality video starts with a green screen. However, if an object passes around the actor / subject, you want the object to remain visible instead of being covered by the actor / subject. Therefore, in addition to the typical requirements for a green screen video, a mixed reality video generally requires a foreground layer.
Fortunately, Fantastic Contraption’s spectator mode can display a separate foreground layer, precisely for mixed reality videos.
Northway Games, the developer of Fantastic Contraption, posted their instructions for mixed reality here. Among other things, it will allow you to do a mixed reality livestream. However, their method requires a webcam or a camera that stream video to a PC. I didn’t want to get a webcam, which would either have mediocre image quality or would have good image quality but would be expensive. I also didn’t want to get an HDMI capture card just to try this out. Hence, here’s a way to do it without a capture card.
WHAT YOU NEED
1. A VR game that has a mixed reality spectator mode. At this time, the only game that has it to my knowledge is Fantastic Contraption but more will follow. (Job Simulator also has a spectator mode but to my knowledge, it currently doesn’t have a foreground layer yet.)
2. Any kind of camera that can record video and has a wide angle and is not a fisheye lens. In my case, I used an Olympus Stylus 1 point-and-shoot, which at its widest is 28mm (in 35mm full frame equivalent terms). A camera with a wider angle lens would have worked even better.
3. A green screen, and a background stand. The background stand I used was the Linco Zenith. I got a 10-foot green chromakey muslin fabric from eBay, although a wider greenscreen would have worked better.
4. A video editor with multitrack / multiple layers and chromakey support. I used Magix Movie Edit Pro 2013 Plus.
5. OBS Studio or other software with screen recording capabilities.
6. Recommended: lights to illuminate your subject. I used a 160-bulb LED light. I also recommend getting an AC adapter for it.
PART 1: RECORDING
1. Setup your greenscreen and your lights.
2. Setup your camera. Use a wide angle and frame it so that the subject will be framed as you intend. Set it close to the play area.
3. Launch Fantastic Contraption. In the settings screen, enable director controls. Under Director Controls, click the dropdown under ‘reality camera’ and select Quartered Display.
 |
| image by Northway Games |
4. By default, the Quartered Display will show three views: on the upper left quadrant is the view from the virtual camera. The upper right quadrant is the same view from the virtual camera, but showing only the foreground layer.
It appears that the location of the controllers determines what is the foreground. Anything between the controller and the virtual camera will be considered ‘foreground’. On the bottom right quadrant is the player’s perspective. Strictly speaking, you don’t need it for the mixed reality video unless you want to switch between a third person and first person perspective.
5. Wear the VR headset and look for the virtual camera. In the case of Fantastic Contraption, it looks like a winged Gear 360 or eyeball (you can see it on the bottom right quadrant in the screenshot in step 4). Move the virtual camera to the space where your real world camera is (you’ll need to peek through the headset from time to time). Point the virtual camera to the same direction as the real world camera.
In the upper quadrant of the quartered view, you should now see the controllers from the same perspective as seen from the real world camera.
6. In the Director Mode controls, set the 3rd person FOV (field of view) to your desired view. It should not be smaller than the field of view visible on your real world camera, but it can be bigger.
7. Run OBS Studio, and under Sources, add Window Capture. In the next screen, click on Create New. In the following screen, under “Window”, click the dropdown box and look for Fanstastic Contraption. Uncheck “Capture Cursor.”
8. Click on “Start Recording” on OBS. Start recording on your real world camera as well. I recommend clapping loudly once to help sync the videos later using audio sync.
9. When you’re done, click on Stop Recording on OBS, and stop recording in your real world camera as well.
PART 2: PROCESSING THE VIDEO
1. From your video editor, import the recorded video from OBS. Add the video to Track 1. Crop it to the upper left quadrant (if percentages are available, specify 50% for height and width). This will be your background.
2. Import the recorded video again from OBS. Add it to Track 3. Crop it to the upper right quadrant. This will be your foreground.
3. Select the foreground video (Track 3). Use the chromakey function of your video editor to remove the background (in this case, I selected Blue then adjusted the threshold until the background disappeared).
4. Adjust the position of Track 3, to align Track 3 with Track 1 exactly. (If you have a ‘centered’ command for both Track 3 and Track 1, you can use that to align them.)
5. Import your real world camera video and add it to Track 2.
6. Sync Track 1, 2 and 3 using the loud clap to help you sync them. If necessary, generate a waveform.
7. Select Track 2 (the real world video). If your greenscreen did not cover the entire width of the frame, crop the video to the edge of the green screen, so that the background of the subject is filled with the greenscreen.
8. Still selecting Track 2, use the chromakey function of your editor to remove the background.
9. Use trial and error to adjust the size and position of Track 2 to match the size and position of the controllers. Even if the resized Track 2 is just a portion of the video from Track 1 and 3, that’s ok.
Congratulations! Your mixed reality video should now be ready for export!
Add Comment