How to create holograms for Looking Glass (quilt format) from a video captured with a phone

Overview

This guide shows how to create holograms for Looking Glass Portrait glasses-free 3D displays. This is achieved by creating a photorealistic 3D model from a video captured with a phone (or potentially any video), then rendering it into the Looking Glass quilt format with Volurama. Volurama is an advanced tool for creating neural radiance fields (NeRFs), which are photorealistic 3D models. This tutorial will show you how to capture the necessary video with a phone, render a holographic "quilt" video for Looking Glass, and add some 3D camera motion to enhance the effect.

Step-by-step instructions

1. Capture an input video with a phone
   Volurama can potentially work with any type of input video, but for reliable results, we suggest starting with the capture process here. We recommend using the widest FOV setting available on the camera. For best results, use a video that is about 5 to 30 seconds long. Move the camera around in a spiral, circle, or square pattern to get each part of the scene from several different points of view. You can capture in portrait or landscape view.
2. Create a new project in Volurama
   Launch Volurama and click "Create New Project".

3. Select an input video
   Select a video file on your computer. It can be an mp4, mov, or mkv file.

4. Select a project directory
   Select a folder where files related to this project will be saved. You should create a new directory just for this. Large amounts of data may be created in this folder, and files may be automatically deleted from it as well.

5. SFM & NeRF options
   This screen allows you to configure settings which affect the quality of the results, and how long it takes to process. For a faster run, leave these at the default values. When ready, click "Start Processing".
   
   Tip: when rendering holograms for Looking Glass, there is little benefit to using HD settings; default settings are faster and look about the same on the Looking Glass Portrait.
   Tip: change "Structure from Motion" ▸ "# Iterations" from 40 to 25 to save time without greatly affecting quality.
   Tip: for videos which contain a lot of moving objects, try changing "Structure from Motion" ▸ "Outlier Percentile" from 0.8 to a smaller number like 0.5 or 0.25.

6. Wait for processing, visualization
   It may take a while to process, depending on your settings and computer. While you wait, there are three main visualizations to look at: keypoint tracking, structure from motion optimization, and NeRF optimization. Keypoint tracking is the first step in the computer vision pipeline for determining camera motion. Structure from motion is the part of the system which solves a math problem to determine the cameras position and orientation in every frame of the input video. NeRF optimization is the part where it uses machine learning to create a photorealistic 3D model of the scene.

7. Real-time 3D preview
   The 3D view is a quick and dirty preview of your scene, not the most photorealistic rendering possible (see the "Preview Render" window for that, and make it bigger). The 3D view only includes nearby objects, it doesn't include the background; its OK if it is missing some parts of the scene, as long as they appear in the "Preview Render" view. The spiral in the video below is the software's estimate of the 3D path the camera followed in the input video.
8. Adjust timeline duration (optional)
   Volurama renders videos as output, and the length of these videos is determined by the timeline. To change the duration of rendered videos, use the menu: Virtual Camera ▸ Change Timeline Duration. Modify the timeline duration first, before setting up virtual camera motion.

9. Virtual camera motion presets
   To create a visually pleasing result, Volurama can render the scene from the point of view of a virtual camera, which moves along any path we want. The simplest way to set up this motion is to use the motion presets, e.g. the menu Virtual Camera ▸ Dolly Forward (or any of the others). Most of these presets create two keyframes in the timeline, one at the first frame and one at the last. It is convenient to start with a preset like this, then modify the keyframes to further adjust the virtual camera motion.

10. Edit keyframes
    Use the Keyframe Editor window to adjust the position and rotation of the virtual camera. The Preview Render window shows what the final output from the virtual camera will be.

11. Virtual camera settings - optimized for Looking Glass Portraint
    To create holographic content, use the following Virtual Camera Settings: the camera type should be "Looking Glass Portrait". The horizontal field of view is an artistic choice. Stereo Baseline determines how far apart the virtual cameras are (for the Looking Glass quilt, this is the distance between the cameras corresponding to the first and last image). Stereo Baseline is not given with any particular units. It is also an artistic choice, which affects how much depth the hologram has.

12. Render video
    To begin rendering the final output, use the menu: File ▸ Render Video. This will open the render config screen, which has the same options as the Virtual Camera Settings, as well as options for which video compression formats to create. For Looking Glass Portrait, the video must be encoded with h264.
13. View rendered frames
    While rendering, each frame is saved as an image in the /render_frames subdirectory of the project. It is useful to look at these results before the full render completes. In fact, you can load individual frame images onto the Looking Glass instead of or before loading the full video. After all frames are finished rendering, a video .mp4 or .mov file is generated in the project directory.
14. Load the quilt on Looking Glass
    After the video finishes rendering (or with a single frame photo), the final step is to load the quilt onto the Looking Glass. To do this, you MUST use Looking Glass Studio. It will not work to simply copy a quilt file to the Looking Glass hard drive. When importing the quilts with Looking Glass Studio, you may need to specify that they have 8 columns and 6 rows. You can give the software a clue about this by renaming the video file to end in '_qs8x6a0.75.mp4'. Refer to the Looking Glass Quilt documentation for more details.
15. The result: a holographic quilt, ready to load on the Looking Glass Portait