Capturing the shape of a dynamic world - fast !

Marcus Magnor, Hans‐Peter Seidel, Myungsoo Kim

Abstract

Acquiring on-line the evolving shape of a dynamic scene from a handful of video streams may be considered one of the most challenging, but at the same time also most auspicious tasks in contemporary computer graphics and computer vision research. The anticipation of revolutionary new applications such as interactive 3D television broadcasts motivates the ongoing work on free-viewpoint video rendering. This paper aims at giving a state-of-progress report on this lively research endeavour. Different acquisition setups and on-line reconstruction approaches are exemplified. Yielding interactive frame rates, depth map-based techniques, polyhedral as well as volumetric visual hull reconstruction approaches, and combined methods employing visual hull-guided depth map estimation are presented. The experience gained with these approaches allows identifying future research directions towards real-time analysis and high-quality synthesis of dynamic, real-world scenes.

Introduction

In the early 2000s, computer graphics excelled at rendering static 3D models, but integrating the "real world" into virtual environments in real-time remained a significant challenge. Pure Image-Based Rendering (IBR) typically requires a dense array of cameras, while traditional 3D reconstruction methods like dense stereo are often too computationally intensive for live applications. This research addresses the challenge of on-line reconstruction: generating high-quality 3D representations from a sparse set of video streams (e.g., six cameras) at speeds suitable for interactive 3D-TV, sports analysis, and immersive gaming.

System Architecture and Distributed Processing

A key insight of this work is that real-time performance necessitates a distributed computing approach. The system architecture offloads heavy computational tasks, such as silhouette extraction and partial volume reconstruction, to dedicated "sensor nodes" (individual PCs for each camera pair).

System architecture showing the flow from sensors through sensor nodes to a central display node and the final viewer.

As shown in the architecture diagram, these nodes send compressed data to a central "display node." This node aggregates the information to form the final 3D model, which is then presented to the viewer. This pipeline allows the system to maintain interactive frame rates by parallelizing the most demanding processing steps.

Reconstruction Methodologies

The paper evaluates four distinct architectural approaches to 3D reconstruction:

  1. Hierarchical Depth Map Estimation (Lumi-Shelf): Utilizing three stereo-camera pairs, this system calculates per-pixel depth. Efficiency is gained by processing only a "Region of Interest" (ROI) and employing hierarchical block-matching.
  2. Real-Time Polyhedral Visual Hulls: This method identifies object silhouettes via background subtraction. These silhouettes are back-projected into 3D space as "cones," and their intersection forms a 3D polyhedron.
  3. Volumetric Visual Hulls: The scene is discretized into a voxel grid. Voxels falling inside all silhouettes are identified as part of the object and rendered using GPU-accelerated "billboards."
  4. Hybrid Visual Hull-Guided Depth Estimation: This sophisticated approach uses the visual hull to define a conservative "search space" for stereo matching. By pruning empty space, the system speeds up depth estimation and captures concavities that standard visual hulls miss.
The process of polyhedral visual hull reconstruction: from input image and silhouette to back-projected cones and the final 3D intersection.

The polyhedral approach, illustrated above, is particularly efficient, achieving rendering speeds of up to 27 fps on contemporary consumer hardware—surpassing the capture rate of the cameras themselves.

Key Results and Performance

The researchers tested these methods using custom hardware setups: "Lumi-Shelf" (three camera pairs) and "The Stage" (six distributed cameras).

  • Polyhedral Hulls: Achieved 27 fps rendering on an NVIDIA GeForce3.
  • Volumetric Hulls: Reconstructed at 7 fps for a $64^3$ grid, with rendering reaching 30 fps at lower resolutions.
  • Hybrid Approach: Produced the highest quality by capturing concavities, though at a slower reconstruction rate of 4 fps.

High visual quality was maintained across methods by using view-dependent texturing, which blends images from the cameras closest to the user's virtual viewpoint to mask minor geometric inaccuracies.

Limitations and Future Directions

Despite these advancements, challenges remain. Deep concavities are difficult for silhouette-based methods to resolve perfectly. Furthermore, the current implementation treats frames independently; incorporating temporal coherence (tracking objects over time) could reduce noise and smooth reconstructions. Future work also aims to estimate surface reflection properties to move beyond the assumption of matte (Lambertian) surfaces, enabling more realistic relighting in virtual scenes.

Summary updated

Abstract

Acquiring on-line the evolving shape of a dynamic scene from a handful of video streams may be considered one of the most challenging, but at the same time also most auspicious tasks in contemporary computer graphics and computer vision research. The anticipation of revolutionary new applications such as interactive 3D television broadcasts motivates the ongoing work on free-viewpoint video rendering. This paper aims at giving a state-of-progress report on this lively research endeavour. Different acquisition setups and on-line reconstruction approaches are exemplified. Yielding interactive frame rates, depth map-based techniques, polyhedral as well as volumetric visual hull reconstruction approaches, and combined methods employing visual hull-guided depth map estimation are presented. The experience gained with these approaches allows identifying future research directions towards real-time analysis and high-quality synthesis of dynamic, real-world scenes.
BibTeX
@inproceedings{magnor2003capturing,
  author = {Marcus Magnor and Hans‐Peter Seidel and Myungsoo Kim},
  title = {{Capturing the shape of a dynamic world - fast !}},
  booktitle = {MPG.PuRe},
  year = {2003},
}
Figure