Mixed Reality Capture

Mixed reality capture lets you record the VR session from a virtual camera in combination with matching footage from a physical camera in the real world. This requires matching the virtual camera’s position, orientation and focal length to the real camera, so that the footage from both will line up correctly.

Set Up For Capture

To match the virtual desktop camera to the real camera, first attach the virtual camera to one of your VR controllers:

1.   Select Cameras > Lock Camera to VR Controller.
2.   Attach the camera to a controller by squeezing the grip buttons on the controller.

Note:  Alternatively you can attach the camera to any tracked VR device (for example a Vive Tracker) by selecting Cameras > Lock Camera and choosing a device.

3.   Place the device which the virtual camera is attached to as close as possible to your real camera. (It can be next to it, on top of it, or generally anywhere near it.)
4.   Select View > Mixed Reality Capture Setup to display the setup panel. This panel allows you to configure settings of the virtual desktop camera, to get it to match your real camera as closely as possible.

5.   Measure the distance and angles between the real camera and the controller. The point to measure from on the controller is the point just above the menu button, where the body of the controller meets the tracking loop. The point to measure to on the camera is the center of the lens (or as close to it as you can get).
6.   Enter these measurements into the Offset from tracker fields.
7.   Set the dimensions of the desktop window to match the resolution of your real camera recording.

Tip:  Alternatively you can use smaller or larger dimensions than your camera, so long as the aspect ratio matches your camera; however that will mean you have to rescale the virtual camera footage when compositing with the real camera footage.

Enter the Width and Height and click the Apply button to resize the window.

8.   Finally, set the field of view on the virtual desktop camera to match your real camera. You can enter the horizontal or vertical angle directly. If you don't know these angles, you can enter your camera’s sensor width and current focal length to have them calculated for you.

All of the values in the Mixed reality capture setup panel can be tweaked interactively using the keys below. Place the cursor in the field to edit its value.

Up Arrow

+1

Down Arrow

-1

Ctrl + Up Arrow

+0.1

Ctrl - Down Arrow

-0.1

Page Up

+10

Page Down

-10

Capturing

1.   Select View > Mixed reality capture setup again to hide the panel. If the sculpt tools panel is visible, hide that too. You should now just have the 3D viewport showing.
2.   Start up your preferred screen recording software and set it up to record the Kanova window.
3.   You can now start recording on both the desktop window and the real camera, then go into VR and do your thing.

Once you’ve finished you will most likely still need to do some work to match up the start points of the virtual and real footage, to remove lens distortion from the real footage and so on. Nuke, also by the Foundry, is an excellent choice for that.