Shape-from-Mask: A Deep Learning Based Human Body Shape Reconstruction from Binary Mask Images

Zhongping Ji, Qi Xiao, Yigang Wang, Gang Xu, Peng Du, Qing Wu

Abstract

3D human body reconstruction is a fundamental task in computer graphics, yet traditional methods often struggle with privacy concerns, clothing occlusions, and the high cost of manual modeling. This paper presents Shape-from-Mask, a novel deep learning framework that reconstructs high-fidelity 3D human shapes from just two 2D binary masks (frontal and lateral views). By combining a dual-branch CNN architecture with aggressive data augmentation, the system generates accurate digital avatars quickly and robustly.

The Challenge of 3D Human Modeling

Creating a 3D digital human typically requires expert manual labor or expensive 3D scanning hardware. While automated modeling from 2D photos has progressed, it faces two major hurdles:

  1. Privacy and Occlusion: RGB images can compromise privacy and often hide the true body shape under clothing.
  2. Annotation Labor: Most models require dense labels, like joint locations or pixel-level segmentations, which are difficult to produce at scale.

Shape-from-Mask addresses these by using only binary silhouettes, which are privacy-preserving, ignore clothing textures, and can even be derived from simple sketches.

The Shape-from-Mask Pipeline

The core of the system is JointMaskNet, which operates in three stages:

Building a Shape Space

Instead of predicting thousands of individual 3D vertex positions, the researchers used Principal Component Analysis (PCA) on a large dataset of human scans. This compresses the complexity of a 3D body into a manageable 50-dimensional vector ($\phi_s$). The neural network's task is reduced to predicting these 50 parameters.

Dual-Branch Architecture

The network employs two separate branches—FrontalNet and LateralNet—based on the DenseNet architecture. DenseNet's feature-reuse capability is particularly effective at extracting geometric cues from simple silhouettes.

Decoupled Training

To prevent the model from becoming "confused" by different perspectives, the frontal and lateral branches are first trained independently. Once each branch becomes an expert in its specific view, they are fused in a joint network to produce the final 3D estimate.

Overcoming Data Scarcity

Deep learning requires massive datasets, but high-quality 3D human scans are rare. The authors expanded the CAESAR database from 4,308 scans to approximately 50,000 models using three strategies: linear interpolation, GAN-based generation, and a novel Segment-and-Merge approach.

The Segment-and-Merge augmentation technique decomposes bodies into parts to create new training samples.

In Segment-and-Merge, bodies are decomposed into parts (arms, legs, torso) and reassembled in different ratios—such as placing long legs on a shorter torso—to create realistic new individuals.

Key Results

The JointMaskNet significantly outperformed single-view networks, achieving much lower Root Mean Square Error (RMSE) in vertex positions. The system is remarkably robust; because it is constrained by a "valid" 3D shape space, it can "fix" noisy or incomplete masks by inferring the most plausible 3D body that fits the outline.

Qualitative results showing the variety and realism of reconstructed 3D human bodies.

Furthermore, the reconstruction is nearly instantaneous once trained, making it ideal for real-time applications like virtual fitting rooms, gaming, and online fashion.

Conclusion and Future Work

Shape-from-Mask demonstrates that binary silhouettes contain rich geometric information sufficient for high-quality 3D reconstruction. While currently limited to standard "A-poses" and medium resolution, future work aims to extend the framework to arbitrary poses and higher-resolution details, potentially integrating more sophisticated generative models.

Summary updated

Abstract

3D content creation is referred to as one of the most fundamental tasks of computer graphics. And many 3D modeling algorithms from 2D images or curves have been developed over the past several decades. Designers are allowed to align some conceptual images or sketch some suggestive curves, from front, side, and top views, and then use them as references in constructing a 3D model automatically or manually. However, to the best of our knowledge, no studies have investigated on 3D human body reconstruction in a similar manner. In this paper, we propose a deep learning based reconstruction of 3D human body shape from 2D orthographic views. A novel CNN-based regression network, with two branches corresponding to frontal and lateral views respectively, is designed for estimating 3D human body shape from 2D mask images. We train our networks separately to decouple the feature descriptors which encode the body parameters from different views, and fuse them to estimate an accurate human body shape. In addition, to overcome the shortage of training data required for this purpose, we propose some significantly data augmentation schemes for 3D human body shapes, which can be used to promote further research on this topic. Extensive experimen- tal results demonstrate that visually realistic and accurate reconstructions can be achieved effectively using our algorithm. Requiring only binary mask images, our method can help users create their own digital avatars quickly, and also make it easy to create digital human body for 3D game, virtual reality, online fashion shopping.
BibTeX
@misc{ji2018shape,
  author = {Zhongping Ji and Qi Xiao and Yigang Wang and Gang Xu and Peng Du and Qing Wu},
  title = {{Shape-from-Mask: A Deep Learning Based Human Body Shape Reconstruction from Binary Mask Images}},
  year = {2018},
  eprint = {1806.08485},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.08485},
}

Loading related content...

Figure