Free-viewpoint video of human actors

Joel Carranza, Christian Theobalt, Marcus A. Magnor, Hans-Peter SeidelACM Trans. Graphics 2003 · DOI · OpenReview

TL;DR

This paper introduces a system for creating "free-viewpoint video" of human actors, allowing viewers to choose their own camera perspective in real-time. By combining marker-free motion capture from synchronized multi-view video with advanced multi-view texturing, the system generates highly realistic and interactive visualizations of human performances.

What Problem Does This Paper Address?

Traditional video and film offer a fixed, 2D perspective, limiting viewer immersion. While computer graphics can render synthetic scenes from any viewpoint, and computer vision can analyze real-world scenes, bridging these to create dynamic, interactive views of real people has been a challenge. Specifically, capturing and rendering realistic human motion from arbitrary viewpoints without intrusive markers or overly complex setups is difficult. Existing methods often rely on expensive hardware, struggle with complex motions, or produce less naturalistic results. This paper aims to provide a robust, non-intrusive system for free-viewpoint video of human actors that is both computationally efficient and visually compelling.

Methods

The core of the system relies on two main components:

  • Marker-Free Motion Capture: The system uses synchronized video footage from multiple cameras. It extracts human silhouettes from these images via background segmentation. These silhouettes are then used to drive a generic 3D human body model. The motion parameters (joint rotations and segment transformations) are estimated offline by optimizing the overlap between the projected model silhouettes and the actual silhouettes from the input images. This approach is non-intrusive, as the actor doesn't need to wear any markers. The optimization is performed in a hierarchical manner, first estimating global body pose and then refining limb and joint parameters, making it robust to complex and fast motions.
  • Multi-View Texturing and Rendering: Once the motion is captured and the 3D body model is posed for each frame, the system generates realistic surface textures. It uses multi-view texturing, blending information from all available camera views onto the 3D model. A novel visibility calculation is employed to handle occlusions and ensure correct texture projection. The rendering subsystem runs in real-time using commodity graphics hardware, allowing for interactive viewpoint manipulation.

Datasets

Not discussed in the paper. The authors mention testing their system on "several multi-view video streams including a male ballet dancer and a second test subject."

Key Insights and Ideas

  • Silhouette-based motion capture is robust and non-intrusive: By focusing on silhouette overlap rather than feature tracking, the system avoids common pitfalls of marker-based or feature-based methods, making it suitable for real-world performances where intrusion is undesirable.
  • Hierarchical optimization for motion capture: Breaking down the motion estimation into a sequence of optimizations (global pose, then limbs, then joints) significantly improves robustness and allows for tracking complex, fast movements that would challenge global optimization.
  • Multi-view texture blending with modified visibility: The approach to handle texture generation, especially around occluded areas, by extending per-vertex visibility to include slightly displaced views is crucial for achieving seamless and realistic surface appearance.
  • Lambertian reflectance for naturalistic texture: While view-dependent texturing can be more realistic, the paper argues that a time-dependent Lambertian texture provides more visually satisfying results and preserves high spatial frequencies, especially when the model geometry isn't perfectly exact.
  • Leveraging GPU for efficiency: Both the motion capture (energy function evaluation) and rendering stages are designed to take advantage of programmable graphics hardware, enabling real-time performance.

Key Results

  • The system successfully tracks human motion, including fast arm movements, complex twisted poses, and full body turns, as demonstrated with a ballet dancer.
  • It can reproduce the actor's appearance with high detail and naturalistic impression from arbitrary viewpoints.
  • The rendering subsystem achieves real-time frame rates (up to 15 fps, with potential for 30 fps) on standard PC hardware.
  • The system is capable of producing high-quality 3D representations with a relatively small number of cameras (eight mentioned).
  • Texture projection artifacts are effectively removed, and subtle details like wrinkles and facial expressions are preserved.

Limitations and Future Work

  • Offline Motion Estimation: The current motion estimation process is done offline, making it unsuitable for live broadcast applications. The authors suggest future work on faster optimization schemes, potentially distributing computation across multiple GPUs.
  • Lambertian Assumption: The system implicitly assumes Lambertian reflectance for texture generation. While effective for skin and many garments, replicating view-dependent reflection effects for even greater realism is a future research direction. This would likely require more precise geometry or advanced techniques like Bidirectional Texture Functions (BTFs).
  • Loose Garments: The system's ability to handle loose garments is mentioned as an area for future research.
  • Extrapolation of Texture: Extrapolating texture for body regions momentarily not visible in any camera view is another area for future work.
  • Relighting: Incorporating known lighting conditions for relighting is also suggested.

Summary updated

Abstract

In free-viewpoint video, the viewer can interactively choose his viewpoint in 3-D space to observe the action of a dynamic real-world scene from arbitrary perspectives. The human body and its motion plays a central role in most visual media and its structure can be exploited for robust motion estimation and efficient visualization. This paper describes a system that uses multi-view synchronized video footage of an actor's performance to estimate motion parameters and to interactively re-render the actor's appearance from any viewpoint.The actor's silhouettes are extracted from synchronized video frames via background segmentation and then used to determine a sequence of poses for a 3D human body model. By employing multi-view texturing during rendering, time-dependent changes in the body surface are reproduced in high detail. The motion capture subsystem runs offline, is non-intrusive, yields robust motion parameter estimates, and can cope with a broad range of motion. The rendering subsystem runs at real-time frame rates using ubiquous graphics hardware, yielding a highly naturalistic impression of the actor. The actor can be placed in virtual environments to create composite dynamic scenes. Free-viewpoint video allows the creation of camera fly-throughs or viewing the action interactively from arbitrary perspectives.
BibTeX
@article{carranza2003free,
  author = {Joel Carranza and Christian Theobalt and Marcus A. Magnor and Hans-Peter Seidel},
  title = {{Free-viewpoint video of human actors}},
  journal = {ACM Trans. Graphics},
  year = {2003},
  doi = {10.1145/882262.882309},
}
Figure