Stereoscopic augmented reality with pseudo-realistic global illumination effects

François de Sorbier, Hideo SaitoSD&A 2014 · DOI

TL;DR

This paper presents a real-time augmented reality system that integrates virtual objects into real-world scenes with convincing global illumination effects. By dynamically capturing scene geometry and light sources using a Kinect depth camera, the system achieves pseudo-realistic rendering, including reflections, refractions, and shadows, without requiring pre-computation or scene reconstruction.

What Problem Does This Paper Address?

Augmented reality (AR) has become increasingly popular, but a significant challenge remains in making virtual objects appear truly integrated and realistic within real-world environments. A key reason for this lack of realism is the difficulty in accurately capturing the scene's geometry and illumination properties in real-time. Most existing AR applications struggle to account for how light interacts with both real and virtual objects, leading to a disconnect. This paper tackles this problem by proposing a system that dynamically estimates scene geometry and light sources to enable realistic rendering of virtual objects, specifically aiming for stereoscopic displays.

Methods

The core of the proposed system relies on a Kinect-like depth camera to capture both color and depth information. Instead of building a full 3D reconstruction of the scene, which can be computationally expensive and slow to update, the authors employ a dynamic environment map. This map is essentially a collection of textures on the faces of a bounding box that surrounds each virtual object.

Here's a breakdown of the key methods:

  • Camera Tracking: The system uses a point-to-plane ICP algorithm, similar to Kinect Fusion, for robust and accurate real-time camera pose estimation. They also incorporate a frame accumulation approach to improve depth map quality and reduce memory usage.
  • Environment Map Update: For each new camera frame, the 3D points from the depth camera are transformed into a global reference frame. These points are then used to update the color and depth information on the faces of the bounding box environment map. This allows the system to capture the local scene geometry and appearance.
  • Light Source Detection: The system identifies potential light sources by analyzing the HSV color representation of the environment map faces. Pixels with high saturation levels, filtered by distance to avoid reflections, are considered candidates. These areas are then segmented into cells, and if a cell contains enough saturated pixels, it's treated as a light source. The center of gravity of these saturated pixels is used to estimate the light source's 3D position.
  • Pseudo-Realistic Rendering:
    • Illumination and Shadows: The system uses Lambertian and Phong illumination models to render virtual objects, influenced by the detected light sources. Shadows are cast onto the scene by employing a ray-casting approach. For each pixel, rays are cast towards the detected light sources. If an intersection with a virtual object is found, the corresponding pixel in the real scene is shadowed. To achieve soft shadows, multiple secondary rays are emitted around the primary ray direction.
  • Reflections and Refractions: The environment map is used to render mirror reflections and refractions on virtual objects. For each vertex of a virtual object, reflection and refraction vectors are computed, and these are used to sample colors from the environment map, creating a sense of the object being integrated into the scene's lighting.
  • Stereoscopic Rendering: The system generates multiple views for stereoscopic display by using the 3D reconstruction from the depth camera to create a triangular mesh. This mesh is then rendered from slightly different viewpoints to produce the input images for stereoscopic devices.

Datasets

Not discussed in the paper. The experiments were conducted in an indoor environment with existing lighting and surfaces.

Key Insights and Ideas

  • Dynamic Environment Maps as a Lightweight Alternative: The paper demonstrates that a dynamic environment map, stored on the faces of a bounding box, can effectively capture scene information for realistic rendering without the heavy computational cost of full 3D scene reconstruction. This is crucial for real-time AR.
  • Light Source Detection from Saturation: Leveraging high saturation in HSV color space as a proxy for light sources is an intuitive and efficient method for estimating illumination in real-time, especially when combined with depth information to filter out reflections.
  • Pseudo-Realism is Achievable with Focused Effects: By focusing on key global illumination effects like diffuse/specular reflections, refractions, and soft shadows, the system achieves a convincing level of realism without needing to implement computationally prohibitive full global illumination techniques.
  • Kinect as a Catalyst for Accessible AR: The paper highlights how affordable depth cameras like the Kinect have democratized research in this area, enabling more sophisticated AR applications.

Key Results

The system achieves a frame rate of 26-30 frames per second with environment map updates and light source computation activated. The average speed for each step of the process is detailed in Table 1:

  • Tracking: 15 msec
  • Update environment map: 5 msec
  • Compute light position: 6 msec
  • Compute shadows: 9 msec
  • Render background with correct depth map: 4 msec
  • Render virtual object: 3 msec
  • Total: 42 msec

Qualitatively, the paper presents examples of virtual objects (a red sphere) integrated into an indoor scene, exhibiting Lambertian reflectance, shadows, mirror reflections, and refractions. The results show virtual objects casting shadows on real surfaces and reflecting the environment.

Limitations and Future Work

  • Limited Global Illumination: The current rendering model only considers direct illumination from detected light sources, not indirect illumination. This means effects like caustics and multiple reflections are not fully captured, leading to the "pseudo-realistic" designation.
  • Static Light Sources: While the system can theoretically detect moving light sources, it currently considers them static for practical implementation.
  • Environment Map Coverage: Users need to manually move the camera to capture the entire environment for the environment map. The authors suggest adding guides to help users ensure complete coverage.
  • Memory Usage of Kinect Fusion: The voxel-based storing approach of Kinect Fusion, while effective for tracking, requires significant memory. The authors propose investigating their frame accumulation approach for pose estimation to reduce memory requirements.
  • Improving Realism: Future work includes exploring ray tracing for more accurate shadow propagation and potentially other global illumination techniques to further enhance realism.

Summary updated

BibTeX
@inproceedings{sorbier2014stereoscopic,
  author = {François de Sorbier and Hideo Saito},
  title = {{Stereoscopic augmented reality with pseudo-realistic global illumination effects}},
  booktitle = {SD\&A},
  year = {2014},
  doi = {10.1117/12.2039280},
}
Figure