Accurate Human Body Reconstruction for Volumetric Video

Decai Chen, Markus Worchel, Ingo Feldmann, Oliver Schreer, Peter Eisert

Abstract

In this work, we enhance a professional end-to-end volumetric video production pipeline to achieve high-fidelity human body reconstruction using only passive cameras. While current volumetric video approaches estimate depth maps using traditional stereo matching techniques, we introduce and optimize deep learning-based multi-view stereo networks for depth map estimation in the context of professional volumetric video reconstruction. Furthermore, we propose a novel depth map post-processing approach including filtering and fusion, by taking into account photometric confidence, cross-view geometric consistency, foreground masks as well as camera viewing frustums. We show that our method can generate high levels of geometric detail for reconstructed human bodies.

Introduction

Volumetric video, or Free-Viewpoint Video, serves as a foundational technology for immersive AR/VR experiences, enabling users to observe performances from arbitrary perspectives. However, generating high-quality 3D assets remains a significant challenge. State-of-the-art "gold standard" systems typically rely on active sensors, such as structured light or infrared (IR) projectors, which are costly and require complex calibration.

Passive systems utilizing standard RGB cameras offer a more accessible alternative but frequently struggle with textureless regions (e.g., plain clothing) and occlusions. Traditional stereo matching techniques often yield noisy depth maps, resulting in 3D models that lack fine geometric details, such as facial features or individual fingers. This research addresses these limitations by leveraging optimized deep learning architectures to bridge the quality gap in passive reconstruction.

Methodology

The proposed pipeline is built upon a professional production environment featuring 32 RGB cameras. The technical framework consists of several key stages:

The proposed volumetric video reconstruction pipeline, illustrating the flow from input images through MVS depth estimation to final mesh generation.

Optimized MVS Network

The system adapts Vis-MVSNet, a state-of-the-art deep learning architecture. To enhance reconstruction accuracy, the authors transitioned from uniform depth sampling to sampling hypotheses uniformly in inverse depth space (disparity space). This approach is more robust for large-scale scenes. Additionally, a more aggressive "coarse-to-fine" strategy was implemented, reducing the depth search range by a factor of 1/8 (compared to the standard 1/4) to achieve higher resolution.

Robust Post-Processing and Fusion

A four-tier filtering approach is applied to the generated depth maps:

  1. Photometric Confidence: Pixels with low network confidence are removed.
  2. Geometric Consistency: Cross-view checks ensure depth consistency between neighboring cameras.
  3. Foreground Masking: Background subtraction eliminates extraneous noise.
  4. Viewing Frustum Filtering: Points outside the logical capture volume are discarded.

Rather than simple point merging, the filtered depth maps are fused into a Truncated Signed Distance Function (TSDF) volume. This voxel-based representation naturally suppresses noise and outliers, facilitating the generation of "watertight" meshes.

Data and Training Strategy

The network is pre-trained on the DTU dataset and fine-tuned on a novel Human Body Reconstruction (HBR) dataset. The HBR dataset contains approximately 12,000 samples from five distinct performances.

The hybrid training strategy blends synthetic high-frequency details with real studio images to prevent overfitting and improve geometric accuracy.

To overcome the "domain shift" problem—where models trained on synthetic data underperform on real-world footage—the authors developed a Hybrid Training Strategy. This involves blending high-frequency geometric signals from synthetic 3D models with low-frequency information from real studio images. Training on purely synthetic data was found to decrease performance (IoU dropped to 63.92%), whereas the blending approach maintains the realism of studio lighting while teaching the network to recognize fine geometric details.

Results and Discussion

The optimized pipeline was evaluated against industry standards, including Iterative Patch Sweep and CasMVSNet.

  • Accuracy: The method achieved an Intersection over Union (IoU) of 98.88%, outperforming all baseline methods.
  • Geometric Detail: Visual results demonstrate significant improvements in challenging areas. While standard models often fail to resolve hand geometry, the proposed method successfully reconstructs individual fingers and sharp facial contours.
  • Efficiency: The full pipeline processes at approximately 5 minutes per frame on a single workstation (Intel Xeon CPU + NVIDIA TITAN RTX), with potential for further acceleration through multi-GPU parallelization.

Conclusion

This work demonstrates that high-fidelity volumetric video can be achieved using passive RGB cameras by optimizing deep MVS networks and implementing robust post-processing. While the system excels at capturing fine geometry, challenges remain regarding non-Lambertian surfaces, such as highly reflective clothing or eyeglasses. Future research may explore Implicit Neural Radiance Fields (NeRFs) to better handle view-dependent lighting effects and further enhance visual realism.

Summary updated

Abstract

In this work, we enhance a professional end-to-end volumetric video production pipeline to achieve high-fidelity human body reconstruction using only passive cameras. While current volumetric video approaches estimate depth maps using traditional stereo matching techniques, we introduce and optimize deep learning-based multi-view stereo networks for depth map estimation in the context of professional volumetric video reconstruction. Furthermore, we propose a novel depth map post-processing approach including filtering and fusion, by taking into account photometric confidence, cross-view geometric consistency, foreground masks as well as camera viewing frustums. We show that our method can generate high levels of geometric detail for reconstructed human bodies.
BibTeX
@inproceedings{chen2021accurate,
  author = {Decai Chen and Markus Worchel and Ingo Feldmann and Oliver Schreer and Peter Eisert},
  title = {{Accurate Human Body Reconstruction for Volumetric Video}},
  booktitle = {IC3D},
  year = {2021},
  url = {https://openreview.net/forum?id=svPBX-iKpey},
}
Figure