Abstract
3D human body reconstruction is a fundamental task in computer graphics, yet traditional methods often struggle with privacy concerns, clothing occlusions, and the high cost of manual modeling. This paper presents Shape-from-Mask, a novel deep learning framework that reconstructs high-fidelity 3D human shapes from just two 2D binary masks (frontal and lateral views). By combining a dual-branch CNN architecture with aggressive data augmentation, the system generates accurate digital avatars quickly and robustly.
The Challenge of 3D Human Modeling
Creating a 3D digital human typically requires expert manual labor or expensive 3D scanning hardware. While automated modeling from 2D photos has progressed, it faces two major hurdles:
- Privacy and Occlusion: RGB images can compromise privacy and often hide the true body shape under clothing.
- Annotation Labor: Most models require dense labels, like joint locations or pixel-level segmentations, which are difficult to produce at scale.
Shape-from-Mask addresses these by using only binary silhouettes, which are privacy-preserving, ignore clothing textures, and can even be derived from simple sketches.
The Shape-from-Mask Pipeline
The core of the system is JointMaskNet, which operates in three stages:
Building a Shape Space
Instead of predicting thousands of individual 3D vertex positions, the researchers used Principal Component Analysis (PCA) on a large dataset of human scans. This compresses the complexity of a 3D body into a manageable 50-dimensional vector ($\phi_s$). The neural network's task is reduced to predicting these 50 parameters.
Dual-Branch Architecture
The network employs two separate branches—FrontalNet and LateralNet—based on the DenseNet architecture. DenseNet's feature-reuse capability is particularly effective at extracting geometric cues from simple silhouettes.
Decoupled Training
To prevent the model from becoming "confused" by different perspectives, the frontal and lateral branches are first trained independently. Once each branch becomes an expert in its specific view, they are fused in a joint network to produce the final 3D estimate.
Overcoming Data Scarcity
Deep learning requires massive datasets, but high-quality 3D human scans are rare. The authors expanded the CAESAR database from 4,308 scans to approximately 50,000 models using three strategies: linear interpolation, GAN-based generation, and a novel Segment-and-Merge approach.
In Segment-and-Merge, bodies are decomposed into parts (arms, legs, torso) and reassembled in different ratios—such as placing long legs on a shorter torso—to create realistic new individuals.
Key Results
The JointMaskNet significantly outperformed single-view networks, achieving much lower Root Mean Square Error (RMSE) in vertex positions. The system is remarkably robust; because it is constrained by a "valid" 3D shape space, it can "fix" noisy or incomplete masks by inferring the most plausible 3D body that fits the outline.
Furthermore, the reconstruction is nearly instantaneous once trained, making it ideal for real-time applications like virtual fitting rooms, gaming, and online fashion.
Conclusion and Future Work
Shape-from-Mask demonstrates that binary silhouettes contain rich geometric information sufficient for high-quality 3D reconstruction. While currently limited to standard "A-poses" and medium resolution, future work aims to extend the framework to arbitrary poses and higher-resolution details, potentially integrating more sophisticated generative models.
Summary updated