Accurate human body reconstruction for volumetric video

Decai Chen, Markus Worchel, Ingo Feldmann, Oliver Schreer, Peter Eisert

TL;DR

Researchers from the Fraunhofer Heinrich Hertz Institute have developed a high-fidelity volumetric video pipeline that reconstructs human bodies using only passive RGB cameras. By optimizing deep learning-based Multi-View Stereo (MVS) networks and introducing a robust post-processing filter, they achieved significantly higher geometric detail—capturing fine features like fingers and facial contours—without the need for expensive active sensors like infrared projectors.

What Problem Does This Paper Address?

Volumetric video (or Free-Viewpoint Video) is the backbone of immersive AR/VR, allowing users to view a performance from any angle. However, creating high-quality 3D assets is difficult. Current "gold standard" systems often rely on active sensors (like structured light or IR) which are expensive and complex to calibrate.

Passive systems—those using only standard RGB cameras—are more accessible but struggle with "textureless" areas (like plain clothing) and occlusions. Traditional stereo matching often produces noisy depth maps, leading to 3D models that look "blobby" or lose fine details like hands and faces. This paper aims to bridge that quality gap using deep learning.

Methods

The authors built upon a professional production setup consisting of 32 RGB cameras. Their technical contributions fall into three main areas:

  1. Optimized MVS Network: They adapted Vis-MVSNet, a state-of-the-art deep learning architecture. To improve accuracy, they moved away from uniform depth sampling. Instead, they distributed sampling hypotheses uniformly in inverse depth space (disparity space), which is more robust for large-scale scenes. They also implemented a more aggressive "coarse-to-fine" strategy, shrinking the depth search range by a factor of 1/8 (compared to the standard 1/4) to achieve finer resolution.
  2. Hybrid Training Strategy: To solve the "domain shift" problem (where networks trained on synthetic data fail on real-world footage), they created a specialized training pipeline that blends high-frequency signals from synthetic 3D models with low-frequency information from real studio images.
  3. Robust Post-Processing: They proposed a four-tier filtering approach for depth maps:
  4. Photometric Confidence: Removing pixels where the network is "unsure."
    • Geometric Consistency: Ensuring a point seen by one camera matches the depth seen by its neighbors.
  5. Foreground Masking: Using background subtraction to kill noise.
  6. Viewing Frustum Filtering: Removing points that fall outside the logical bounds of the capture volume.
  7. TSDF Fusion: Instead of simply merging points, they fused depth maps into a Truncated Signed Distance Function (TSDF) volume. This mathematical representation naturally suppresses noise and outliers before the final mesh is generated.

Datasets

  • DTU Dataset: Used for initial pre-training of the MVS networks.
  • HBR (Human Body Reconstruction) Dataset: A new dataset created by the authors. It contains ~12,000 samples from five different human performances. It uses a "blended" approach: taking real studio images and augmenting them with synthetic high-frequency details to help the network learn fine geometry without losing the "look" of real-world lighting.

Key Insights and Ideas

  • The "Inverse Depth" Advantage: Sampling along the epipolar line in inverse depth space builds a more discriminative "cost volume." This allows the network to be much more precise in placing surfaces in 3D space.
  • Blending is Better than Pure Synthetic: The authors found that training on purely synthetic images actually decreased performance (IoU dropped to 63.92%). By blending real and synthetic data, they prevented the network from overfitting to "perfect" computer-generated pixels.
  • TSDF vs. Point Fusion: While point-based fusion is common, it often leaves "holes" or noisy artifacts. Fusing into a voxel-based TSDF volume acts as a natural spatial filter, resulting in a "watertight" and smoother mesh.

Key Results

The researchers compared their method against several industry standards, including Iterative Patch Sweep and CasMVSNet.

  • Accuracy: Their optimized pipeline achieved an Intersection over Union (IoU) of 98.88%, the highest among all tested methods.
  • Qualitative Detail: The visual results show a marked improvement in "difficult" areas. While standard models often turned hands into mittens, this method successfully reconstructed individual, distinguishable fingers.
  • Efficiency: Despite the complexity, the full pipeline processes at about 5 minutes per frame on a single machine (Intel Xeon CPU + NVIDIA TITAN RTX), with significant potential for parallelization across multiple GPUs.

Limitations and Future Work

Despite the high fidelity, the authors noted a few remaining hurdles:

  • Non-Lambertian Surfaces: The system still struggles with highly reflective objects, such as eyeglasses or shiny silk, because MVS assumes surfaces reflect light uniformly in all directions.
  • Future Direction: The authors suggest exploring Implicit Neural Radiance Fields (NeRFs) as a potential next step to handle these view-dependent lighting effects and further improve realism.

Summary updated

Abstract

In this work, we enhance a professional end-to-end volumetric video production pipeline to achieve high-fidelity human body reconstruction using only passive cameras. While current volumetric video approaches estimate depth maps using traditional stereo matching techniques, we introduce and optimize deep learning-based multi-view stereo networks for depth map estimation in the context of professional volumetric video reconstruction. Furthermore, we propose a novel depth map post-processing approach including filtering and fusion, by taking into account photometric confidence, cross-view geometric consistency, foreground masks as well as camera viewing frustums. We show that our method can generate high levels of geometric detail for reconstructed human bodies.
BibTeX
@inproceedings{chen2021accurate,
  author = {Decai Chen and Markus Worchel and Ingo Feldmann and Oliver Schreer and Peter Eisert},
  title = {{Accurate human body reconstruction for volumetric video}},
  booktitle = {IC3D},
  year = {2021},
  url = {https://openreview.net/forum?id=svPBX-iKpey},
}

Loading related content...

Figure