3D Human Reconstruction in the Wild with Synthetic Data Using Generative Models

Yongtao Ge, Wenjia Wang, Yongfan Chen, Hao Chen, Chunhua Shen

Abstract

In this work, we show that synthetic data created by generative models is complementary to computer graphics (CG) rendered data for achieving remarkable generalization performance on diverse real-world scenes for 3D human pose and shape estimation (HPS). Specifically, we propose an effective approach based on recent diffusion models, termed HumanWild, which can effortlessly generate human images and corresponding 3D mesh annotations. By training a customized ControlNet on a large-scale human-centric dataset, we generate diverse human images and initial ground-truth labels using surface normal images from SMPL-X. We apply the Segment Anything Model (SAM) to filter noise and ensure high-quality annotations. The resulting dataset comprises 0.79M images covering versatile viewpoints and scenes. Our experiments on benchmarks like 3DPW, RICH, and EgoBody verify that exclusively employing generative models for large-scale data generation eliminates the need for real-world data collection while achieving state-of-the-art results.

Introduction: Bridging the Reality Gap

Estimating the 3D mesh of a human body from a single 2D image—Human Pose and Shape (HPS) estimation—is critical for applications in robotics, digital fashion, and augmented reality. However, training robust models requires massive datasets of images paired with precise 3D ground truth. Traditionally, researchers have relied on indoor Motion Capture (Mocap), which is accurate but lacks environmental diversity, or Computer Graphics (CG) rendering, which is scalable but often fails to capture the visual complexity of real-world "in-the-wild" photos.

This "reality gap" leads to models that struggle with complex lighting, diverse body types, and unusual camera angles. To address this, we introduce HumanWild, a pipeline that leverages the vast visual knowledge of large-scale generative models (like Stable Diffusion) to create synthetic training data that is both visually realistic and perfectly annotated.

The HumanWild Pipeline

The core of our approach is a data generation loop that ensures 3D labels align perfectly with generated pixels. Unlike simple text-to-image models, HumanWild uses a multi-condition ControlNet to maintain structural integrity.

The HumanWild data generation pipeline, integrating LLMs, ControlNet, and SAM for high-fidelity synthetic data.

The process begins by sampling a 3D pose from the AMASS motion library and rendering it into spatial maps, specifically depth and surface normals. An LLM (such as ChatGPT) generates a descriptive text prompt to provide environmental context. These inputs are fed into a customized ControlNet (based on SDXL) to generate the final image. To ensure the highest quality, we employ the Segment Anything Model (SAM) to filter out images where the generated person does not align with the input mask, and use SMPLify to refine the final 3D parameters.

Leveraging Surface Normals for Accuracy

A significant challenge in HPS is "front-back ambiguity," where a model cannot distinguish if a person is facing toward or away from the camera. We found that conditioning the generative model on surface normal maps—which describe the orientation of the body's surface—significantly reduces these errors compared to using keypoints or depth alone.

Examples of generated images alongside their corresponding depth and surface normal maps used for 3D annotation.

This synthetic data is designed to be complementary to traditional CG data. While CG provides structural accuracy, diffusion models provide the "texture" of reality, including authentic clothing folds, realistic lighting, and background diversity. The generated dataset comprises 0.79M images with corresponding 3D annotations, covering versatile viewpoints, scenes, and human identities.

Key Results and Generalization

The HumanWild dataset was evaluated across a wide range of benchmarks to verify its effectiveness:

  • Benchmark Superiority: On the 3DPW dataset, adding HumanWild data to existing synthetic datasets (BEDLAM + AGORA) improved the Procrustes-Aligned Mean Per-Joint Position Error (PA-MPJPE) from 47.4mm to 44.9mm.
  • Handling Perspective Distortion: On the SPEC-MTP benchmark, which features wide-angle lenses and significant distortion, HumanWild outperformed CG-only datasets by a wide margin (90.0mm vs 102.4mm), proving its superior generalization to varied camera setups.
  • Complex Poses: The model showed significant gains on the MOYO dataset, which contains challenging yoga poses rarely found in standard mocap datasets.

Conclusion

HumanWild demonstrates that generative models can effectively bridge the gap between synthetic and real-world data for 3D human reconstruction. By eliminating the need for expensive real-world data collection and providing a scalable way to generate high-quality 3D annotations, this approach sets a new standard for HPS training. Future work will focus on improving multi-human scaling and extending the pipeline to human-object interactions and animal pose estimation.

Summary updated

Abstract

In this work, we show that synthetic data created by generative models is complementary to computer graphics (CG) rendered data for achieving remarkable generalization performance on diverse real-world scenes for 3D human pose and shape estimation (HPS). Specifically, we propose an effective approach based on recent diffusion models, termed HumanWild, which can effortlessly generate human images and corresponding 3D mesh annotations. We first collect a large-scale human-centric dataset with comprehensive annotations, e.g., text captions and surface normal images. Then, we train a customized ControlNet model upon this dataset to generate diverse human images and initial ground-truth labels. At the core of this step is that we can easily obtain numerous surface normal images from a 3D human parametric model, e.g., SMPL-X, by rendering the 3D mesh onto the image plane. As there exists inevitable noise in the initial labels, we then apply an off-the-shelf foundation segmentation model, i.e., SAM, to filter negative data samples. Our data generation pipeline is flexible and customizable to facilitate different real-world tasks, e.g., ego-centric scenes and perspective-distortion scenes. The generated dataset comprises 0.79M images with corresponding 3D annotations, covering versatile viewpoints, scenes, and human identities. We train various HPS regressors on top of the generated data and evaluate them on a wide range of benchmarks (3DPW, RICH, EgoBody, AGORA, SSP-3D) to verify the effectiveness of the generated data. By exclusively employing generative models, we generate large-scale in-the-wild human images and high-quality annotations, eliminating the need for real-world data collection.
BibTeX
@article{ge2024d,
  author = {Yongtao Ge and Wenjia Wang and Yongfan Chen and Hao Chen and Chunhua Shen},
  title = {{3D Human Reconstruction in the Wild with Synthetic Data Using Generative Models}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
}
Figure