Sparse-View 3D Reconstruction of Clothed Humans via Normal Maps

Jane Wu, Diego Thomas, Ronald FedkiwWACV 2025

TL;DR

This paper introduces a novel weakly-supervised approach for reconstructing 3D clothed human geometry from a sparse set of RGB images (even a single image). By leveraging 2D normal maps as supervision and employing a differentiable Marching Tetrahedra algorithm, they achieve state-of-the-art results, democratizing 3D human reconstruction for AR/VR and robotics applications.

What Problem Does This Paper Address?

Reconstructing detailed 3D models of clothed humans from images is a challenging problem. Traditional methods often rely on extensive 3D scanning setups or hundreds of input images, making them impractical for many real-world applications. While recent advances like NeRFs and Gaussian Splatting have shown promise, their heavy reliance on dense multiview data limits their applicability. Furthermore, existing methods trained on controlled environments often struggle to generalize to "in-the-wild" scenarios with diverse people and lower-quality images. This paper aims to bridge this gap by enabling high-quality 3D reconstruction from a sparse set of RGB images, making the technology more accessible.

Methods

The core of the proposed method is a deep learning framework that infers a Signed Distance Function (SDF) for a clothed human body represented by a skinned tetrahedral mesh. Here's a breakdown of the key components:

  • Weak Supervision with Normal Maps: Instead of relying on direct 3D ground truth, the network is trained using 2D normal maps derived from the input images. This is a crucial aspect, as normal maps capture surface orientation, which is a strong signal for 3D geometry.
  • Tetrahedral Mesh Representation: The human body's volume is parameterized by a tetrahedral mesh. The network learns SDF values for the vertices of this mesh.
  • Differentiable Marching Tetrahedra: A key innovation is the use of the Marching Tetrahedra algorithm for generating a triangle mesh from the SDF. The authors derive analytical gradients for this process, enabling end-to-end optimization of the network. This is an improvement over methods that approximate gradients, which can lead to inaccuracies.
  • Differentiable Image Rasterization: The generated triangle mesh is then rendered to produce a normal map. This rasterization process is also made differentiable, allowing gradients to flow back to the SDF.
  • Regularization: To ensure the inferred SDFs are accurate and smooth, the authors employ several regularization terms:
  • Eikonal Regularization: Encourages the SDF to behave like a true SDF, where the gradient norm is 1.
  • Motion by Mean Curvature: Promotes surface smoothness.
  • Silhouette Losses: Shrinking and expanding losses are used to match the predicted silhouette with the ground truth silhouette.
  • Skinning: The tetrahedral mesh is skinned to an estimated human pose (e.g., from SMPL) to produce the final clothed human mesh.

Datasets

The paper evaluates its approach on two datasets:

  • SynWild: A synthetic dataset generated from captured 4D scans of dynamic human subjects, placed in realistic scenes with ground truth 3D information.
  • People Snapshot: A dataset of "in-the-wild" video sequences of subjects in static poses, captured with fixed cameras. For this dataset, the authors use PIFuHD-inferred normal maps as a proxy for ground truth, as direct 3D labels are not available.

Key Insights and Ideas

  • Normal Maps as Powerful Supervision: The paper strongly advocates for using 2D normal maps as a more accessible and effective supervisory signal for 3D reconstruction compared to direct 3D geometry supervision, especially for in-the-wild data.
  • End-to-End Differentiable Meshing: Deriving analytical gradients for Marching Tetrahedra is a significant technical contribution, enabling robust end-to-end optimization of implicit surface representations that are then converted to explicit meshes.
  • Weak Supervision for Generalization: By avoiding direct 3D supervision and focusing on image-based cues, the method shows improved generalization capabilities to unseen scenarios.
  • Efficiency Gains: The proposed method is significantly faster than some prior state-of-the-art methods, making it more practical for real-time applications.

Key Results

  • State-of-the-Art Performance: The method achieves state-of-the-art results on the SynWild dataset, outperforming existing methods in volumetric IoU, Chamfer distance, and normal consistency.
  • Strong Performance on In-the-Wild Data: On the People Snapshot dataset, the approach also outperforms other methods in normal error, even when trained with significantly fewer frames.
  • Speedup: The method is reported to be at least 50x faster than SelfRecon on a NVIDIA 3090 GPU, with training times around 20 minutes compared to over a day for SelfRecon.
  • Ablation Studies: Experiments show that increasing the number of sparse views used for training leads to progressively better reconstruction accuracy, with 5 views yielding highly accurate results.

Limitations and Future Work

  • Reliance on Pre-trained Normal Estimators: While the method is weakly supervised, it still relies on pre-trained networks (like PIFuHD) to infer "ground truth" normal maps for training on datasets without explicit 3D labels.
  • Camera Parameter Estimation: For in-the-wild scenarios with uncalibrated cameras, the paper mentions refining camera parameters iteratively. Errors in camera estimation can still impact reconstruction quality.
  • Detail on Backside: The paper notes that the network can produce less detail on the backside of the mesh when only front-facing views are available.
  • Future Work: The authors suggest that their approach for acquiring ground truth data (via refining camera extrinsics and using normal maps) could be a cheaper alternative to 3D scanning for training future 3D reconstruction networks. They also highlight the potential for further improvements by exploring more sophisticated regularization techniques.

Summary updated

BibTeX
@inproceedings{wu2025sparse,
  author = {Jane Wu and Diego Thomas and Ronald Fedkiw},
  title = {{Sparse-View 3D Reconstruction of Clothed Humans via Normal Maps}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure