Three-Dimensional Human Body Reconstruction Using Dual-View Normal Maps

Yukun Dong, Shengtao Wang, Junqi Sun, Menghua Wang, Long ChengSymmetry 2024 · DOI

TL;DR

This paper introduces a novel method for reconstructing highly detailed 3D human body models using only front and back view images. By leveraging the symmetry between these views and employing a fusion technique with implicit neural representations, the method significantly improves surface detail and accuracy compared to single-view approaches, while being more efficient than multi-view methods.

What Problem Does This Paper Address?

Accurately reconstructing 3D human bodies from images is a long-standing challenge in computer graphics and human-computer interaction. Single-view methods often suffer from distortions and lost details due to self-occlusion, especially on the back of the body. While multi-view approaches can capture more information, they can be computationally expensive and often prioritize pose estimation over detailed shape reconstruction. Existing methods struggle to achieve both high fidelity in surface details and efficient reconstruction, particularly for applications like virtual try-on or realistic avatar creation.

Methods

The proposed method builds upon the SMPL-X parametric human body model. The core innovation lies in its dual-view approach:

  • Dual-View Input: The method takes RGB images from both the front and back views of a subject.
  • Normal and Depth Estimation: Stacked hourglass networks are used to estimate normal maps and depth maps for both the front and back views. A novel contour distance loss is introduced to ensure alignment between the front and back depth maps.
  • Initial SMPL-X Estimation: An initial SMPL-X model is generated from the front view images.
  • Surface Reconstruction and Fusion: The estimated normal and depth maps, along with the initial SMPL-X model, are used to reconstruct front and back 3D surfaces. The d-BiNI method is employed, incorporating a flipped constraint for the back view depth.
  • Implicit Surface Fusion: An implicit function network (IF-Nets+) is used to infer the fusion boundary between the front and back surfaces, creating a seamless model.
  • Poisson Reconstruction: This technique is then used to combine the front surface, back surface, and the inferred boundary into a complete, high-fidelity clothed human model.
  • Parametric Model Optimization: The SMPL-X parameters (shape, pose) are iteratively optimized using the reconstructed front and back surfaces as constraints, further refining the internal body model. This step aims to minimize discrepancies between the SMPL-X mesh and the inferred surfaces.

Datasets

The paper utilizes the following datasets for training and evaluation:

  • THuman2.0: Used for training the normal map estimation and silhouette encoder. A subset was used for validation.
  • RenderPeople: A small selection of scans with loose clothing and complex poses was used for qualitative evaluation of method flexibility.
  • THuman3.0: Used for evaluating reconstruction effects on human-garment combinations.
  • Paired Images (from [16]): A unique collection used for further validation.
  • Custom Dataset: 120 pairs of images from different subjects, with concurrent body metric measurements (waist circumference, chest size, shoulder width), were collected for validation of nude human body models.

Key Insights and Ideas

  • Symmetry as a Constraint: The inherent symmetry between front and back views of the human body is a powerful cue that can significantly reduce ambiguity and improve reconstruction accuracy, especially for occluded regions.
  • Fusion of Parametric and Implicit Models: Combining the structured representation of SMPL-X with the detailed surface generation capabilities of implicit neural networks (like IF-Nets+) allows for both accurate body shape/pose estimation and high-fidelity surface detail.
  • Contour Alignment is Crucial: Explicitly enforcing alignment of depth contours between front and back views is vital for creating a coherent and accurate 3D surface, preventing boundary mismatches.
  • Iterative Refinement: Optimizing the parametric SMPL-X model after initial surface reconstruction, using the reconstructed surfaces as references, leads to more accurate internal body representations that better match the external geometry.

Key Results

  • Quantitative Improvement: The proposed method ("Ours") achieved the lowest reconstruction errors (Chamfer and P2S distances) on the THuman2.0 and RenderPeople datasets compared to several state-of-the-art single-view methods (PIFu-HD, ICON, ECON, D-IF). For instance, on THuman2.0, the Chamfer distance was 1.171 cm, and P2S was 1.399 cm.
  • Body Measurement Accuracy: When validating nude human body models, the method achieved an average measurement error below 0.8 cm for circumference and length measurements across 120 individuals, demonstrating robustness and accuracy for body sizing applications.
  • Qualitative Superiority: Visual comparisons showed that the dual-view approach significantly improves the reconstruction of the back of the body, resolving blurring and inaccuracies often seen in single-view methods. The method produces realistic and detailed texturing, especially on the back.
  • Efficiency: The method is noted to be efficient, generating results in 3-5 seconds on a NVIDIA Tesla V100 GPU, making it suitable for real-time applications.

Limitations and Future Work

  • Dual-View System Dependency: The primary limitation is the requirement for a dual-view camera system, which restricts its application in scenarios where such setups are unavailable.
  • Extreme Postures and Clothing: Complex postures (e.g., curling, extreme twisting) and intricate clothing can still pose challenges for the surface fusion algorithm, potentially leading to inaccuracies.
  • Future Work: The authors plan to extend the framework to support free-view images by integrating diffusion models for plausible back-view synthesis. They also aim to improve posture estimation accuracy and clothing surface completion for extreme cases. Further research into guiding parametric models to drive clothing transformations is also proposed.

Summary updated

BibTeX
@article{dong2024three,
  author = {Yukun Dong and Shengtao Wang and Junqi Sun and Menghua Wang and Long Cheng},
  title = {{Three-Dimensional Human Body Reconstruction Using Dual-View Normal Maps}},
  journal = {Symmetry},
  year = {2024},
  doi = {10.3390/sym16121647},
}
Figure