Production framework for full panoramic scenes with photorealistic augmented reality

Dalai Felinto, Aldo René Zang, Luiz Velho

TL;DR

This paper introduces an end-to-end production framework for merging synthetic 3D elements into 360° panoramas with high physical accuracy. By proposing a novel light-depth environment map—which stores both radiance and spatial depth—the authors enable realistic local lighting, accurate shadow orientation, and physical interactions that traditional image-based lighting (IBL) techniques cannot achieve.

What Problem Does This Paper Address?

In traditional computer graphics, Image-Based Lighting (IBL) uses high-dynamic-range (HDR) panoramas to light synthetic objects. However, these panoramas are typically treated as "directional maps," assuming all light sources are at an infinite distance.

This assumption breaks down in "augmented reality" scenarios where a synthetic object is placed inside a real, captured room. If you place a virtual ball near a real lamp, a standard IBL approach won't know the lamp is close; it will calculate shadows as if the light were coming from infinity. Furthermore, traditional methods struggle with reflections that should transition seamlessly between real and virtual surfaces, and they offer no easy way to handle physical interactions (like a virtual ball "denting" a real sofa). The authors aim to bridge this gap for immersive formats like fulldomes and panoramic cinema.

Methods

The authors provide a complete pipeline from capture to final render:

  1. Environment Capture: Using semi-professional gear (DSLR with a fisheye lens) to capture bracketed photos, which are stitched into an HDR equirectangular panorama.
  2. Calibration & Scaling: A "plane-centric" workflow where the user identifies a known rectangular shape (like a rug or floor section) to establish the world's orientation (horizon) and physical scale (camera height).
  3. Environment Mesh Construction: Using a custom plugin for Blender, the artist creates a rough 3D proxy of the real room. This mesh is "guided" by the panorama.
  4. Light-Depth Map: This is the core innovation. The framework generates a map that stores the distance (depth) to every pixel in the panorama.
  5. ENVPath Integrator: A modified path-tracing algorithm (extending ARLuxrender) that classifies scene elements into three types:
  6. Synthetic: New 3D objects.
  7. Support: Real surfaces that need to receive new shadows/reflections (e.g., the floor).
  8. Environment: Surfaces that only provide light or background.
  9. Advanced Rendering: Because the system knows the position of light sources (not just their direction), it can calculate accurate local shadows and reflections. It also uses "Environment Texture Coordinates" to allow synthetic objects to deform the real environment's textures.

Datasets

The paper does not use standard machine learning datasets. Instead, the authors validated their framework using:

  • Custom Captures: HDR panoramas of office and laboratory environments.
  • Third-Party Data: A panorama from blendedskies.com (Sam Schad) was used to prove the framework's flexibility with external sources.

Key Insights and Ideas

  • Depth is the "Aha!" Moment: By adding a depth channel to a standard HDR panorama, you transform a flat background into a 3D light field. This allows the renderer to treat every pixel in the panorama as a point light source with a specific 3D position.
  • One-Pass Solution: Unlike traditional VFX workflows that require multiple "passes" (beauty, shadow, reflection) and complex compositing, this framework produces the final integrated image in a single render pass.
  • Support vs. Environment Primitives: The distinction between surfaces that receive light (support) and those that emit it (environment) allows the path tracer to intelligently decide when to use captured data and when to calculate new light interactions.

Key Results

  • Accurate Local Shadows: As shown in Figure 15, two balls placed in different parts of a room correctly show different shadow directions based on their proximity to local light sources.
  • Seamless Reflections: Reflections in synthetic chrome spheres correctly transition from reflecting other virtual objects to reflecting the real captured environment with proper perspective.
  • Physical Interaction: The authors demonstrated a "heavy" virtual ball deforming a real couch by using the environment mesh and texture coordinates to warp the captured image of the couch.
  • Camera Traveling: The framework allows for small camera movements (parallax) within the captured scene, which is usually impossible with a static 2D panorama.

Limitations and Future Work

  • Blind Spots: Because the framework relies on a single-point panorama, any area occluded from the camera's original position (e.g., the space behind a table) contains no data. If a virtual object moves into these "blind spots," the illusion breaks.
  • Manual Effort: Reconstructing the environment mesh still requires manual labor by an artist in Blender.
  • Future Directions: The authors suggest using multiple panoramic captures to fill in occlusions and allow for wider camera movement. They also aim to automate the mesh construction process using point clouds from depth sensors like the Kinect.

Summary updated

Abstract


 
 
 The novelty of our proposal is the end-to-end solution to combine computer generated elements and captured panoramas. This framework supports productions specially aimed at spherical dis- plays (e.g., fulldomes). Full panoramas are popular in the computer graphics industry. However their common usage on environment lighting and reflection maps are often restrict to conven- tional displays. With a keen eye in what may be the next trend in the filmmaking industry, we address the particularities of those productions, exploring a new representation of the space by storing the depth together with the light map, in a full panoramic light-depth map.
 
 
BibTeX
@inproceedings{felinto2012production,
  author = {Dalai Felinto and Aldo René Zang and Luiz Velho},
  title = {{Production framework for full panoramic scenes with photorealistic augmented reality}},
  booktitle = {CLEI},
  year = {2012},
  doi = {10.1109/clei.2012.6427123},
}

Loading related content...

Figure