How to create stereoscopic videos for LumePad from photorealistic 3D capture with a phone

Overview

This guide details how to generate stereoscopic 3D animations by creating a photorealistic 3D model of an environment using a short video, possibly captured with a phone. Stereoscopic 3D videos of this kind are ideally suited for viewing on glasses-free 3D displays, such as the LumePad, or through parallel free-viewing. Volurama is an advanced tool for creating neural radiance fields (NeRFs), which are photorealistic 3D models. This tutorial will show you how to capture the necessary video with a phone and configure the output for stereoscopic rendering.

Step-by-step instructions

1. Capture an input video with a phone
   Volurama can potentially work with any type of input video, but for reliable results, we suggest starting with the capture process here. We recommend using the widest FOV setting available on the camera. For best results, use a video that is about 5 to 30 seconds long. Move the camera around in a spiral, circle, or square pattern to get each part of the scene from several different points of view. You can capture in portrait or landscape view.
2. Create a new project in Volurama
   Launch Volurama and click "Create New Project".

3. Select an input video
   Select a video file on your computer. It can be an mp4, mov, or mkv file.

4. Select a project directory
   Select a folder where files related to this project will be saved. You should create a new directory just for this. Large amounts of data may be created in this folder, and files may be automatically deleted from it as well.

5. SFM & NeRF options
   This screen allows you to configure settings which affect the quality of the results, and how long it takes to process. For a faster run, leave these at the default values. If you are planning on rendering HD output (e.g., for LumePad or Spatial Video), it is recommended to press the "HD Settings" button. When ready, click "Start Processing".
   
   TLDR: Just click "HD Settings", then "Start Processing".
   Tip: change "Structure from Motion" ▸ "# Iterations" from 40 to 25 to save time without greatly affecting quality.
   Tip: for videos which contain a lot of moving objects, try changing "Structure from Motion" ▸ "Outlier Percentile" from 0.8 to a smaller number like 0.5 or 0.25.

6. Wait for processing, visualization
   It may take a while to process, depending on your settings and computer. While you wait, there are three main visualizations to look at: keypoint tracking, structure from motion optimization, and NeRF optimization. Keypoint tracking is the first step in the computer vision pipeline for determining camera motion. Structure from motion is the part of the system which solves a math problem to determine the cameras position and orientation in every frame of the input video. NeRF optimization is the part where it uses machine learning to create a photorealistic 3D model of the scene.

7. Real-time 3D preview
   The 3D view is a quick and dirty preview of your scene, not the most photorealistic rendering possible (see the "Preview Render" window for that, and make it bigger). The 3D view only includes nearby objects, it doesn't include the background; its OK if it is missing some parts of the scene, as long as they appear in the "Preview Render" view. The spiral in the video below is the software's estimate of the 3D path the camera followed in the input video.
8. Adjust timeline duration (optional)
   Volurama renders videos as output, and the length of these videos is determined by the timeline. To change the duration of rendered videos, use the menu: Virtual Camera ▸ Change Timeline Duration. Modify the timeline duration first, before setting up virtual camera motion.

9. Virtual camera motion presets
   To create a visually pleasing result, Volurama can render the scene from the point of view of a virtual camera, which moves along any path we want. The simplest way to set up this motion is to use the motion presets, e.g. the menu Virtual Camera ▸ Dolly Forward (or any of the others). Most of these presets create two keyframes in the timeline, one at the first frame and one at the last. It is convenient to start with a preset like this, then modify the keyframes to further adjust the virtual camera motion.

10. Edit keyframes
    Use the Keyframe Editor window to adjust the position and rotation of the virtual camera. The Preview Render window shows what the final output from the virtual camera will be.

11. World transform (optional)
    The World Transform editor lets you rotate the world, which is useful for adjusting the horizon to be level, or for artistic effect.
    
    Tip: it is easier to use the sliders after resizing the World Transform editor window.

12. Virtual camera settings - optimized for LumePad
    To create content for the LumePad by Leia Inc., use the following Virtual Camera Settings: the camera type should be "Rectilinear (Stereo)". The width and height should be at least 1920x1080. Horizontal FOV and Stereo Baseline are artistic choices. Stereo Baseline affects how far appart the virtual left and right cameras are, which changes the viewer's perception of depth and scale.
    
    Tip: resize the Preview Render window for a more detailed preview of the results, and consider closing other editor windows.

13. Render video
    To begin rendering the final output, use the menu: File ▸ Render Video. This will open the render config screen, which has the same options as the Virtual Camera Settings, as well as options for which video compression formats to create. For LumePad, h264 or h265 are OK, and ProRes is preferred for near-lossless video editing.


14. View rendered frames
    While rendering, each frame is saved as an image in the /render_frames subdirectory of the project. It is useful to look at these results before the full render completes. After all frames are finished rendering, a video .mp4 or .mov file is generated in the project directory.

15. The result: a photorealistic 3D video flythrough ready to view on LumePad
    Now we have a stereoscopic flythrough video which is ready to view in 3D on LumePad, or parallel free-viewing.