Abstract
In this work, we show that synthetic data created by generative models is complementary to computer graphics (CG) rendered data for achieving remarkable generalization performance on diverse real-world scenes for 3D human pose and shape estimation (HPS). Specifically, we propose an effective approach based on recent diffusion models, termed HumanWild, which can effortlessly generate human images and corresponding 3D mesh annotations. By training a customized ControlNet on a large-scale human-centric dataset, we generate diverse human images and initial ground-truth labels using surface normal images from SMPL-X. We apply the Segment Anything Model (SAM) to filter noise and ensure high-quality annotations. The resulting dataset comprises 0.79M images covering versatile viewpoints and scenes. Our experiments on benchmarks like 3DPW, RICH, and EgoBody verify that exclusively employing generative models for large-scale data generation eliminates the need for real-world data collection while achieving state-of-the-art results.
Introduction: Bridging the Reality Gap
Estimating the 3D mesh of a human body from a single 2D image—Human Pose and Shape (HPS) estimation—is critical for applications in robotics, digital fashion, and augmented reality. However, training robust models requires massive datasets of images paired with precise 3D ground truth. Traditionally, researchers have relied on indoor Motion Capture (Mocap), which is accurate but lacks environmental diversity, or Computer Graphics (CG) rendering, which is scalable but often fails to capture the visual complexity of real-world "in-the-wild" photos.
This "reality gap" leads to models that struggle with complex lighting, diverse body types, and unusual camera angles. To address this, we introduce HumanWild, a pipeline that leverages the vast visual knowledge of large-scale generative models (like Stable Diffusion) to create synthetic training data that is both visually realistic and perfectly annotated.
The HumanWild Pipeline
The core of our approach is a data generation loop that ensures 3D labels align perfectly with generated pixels. Unlike simple text-to-image models, HumanWild uses a multi-condition ControlNet to maintain structural integrity.
The process begins by sampling a 3D pose from the AMASS motion library and rendering it into spatial maps, specifically depth and surface normals. An LLM (such as ChatGPT) generates a descriptive text prompt to provide environmental context. These inputs are fed into a customized ControlNet (based on SDXL) to generate the final image. To ensure the highest quality, we employ the Segment Anything Model (SAM) to filter out images where the generated person does not align with the input mask, and use SMPLify to refine the final 3D parameters.
Leveraging Surface Normals for Accuracy
A significant challenge in HPS is "front-back ambiguity," where a model cannot distinguish if a person is facing toward or away from the camera. We found that conditioning the generative model on surface normal maps—which describe the orientation of the body's surface—significantly reduces these errors compared to using keypoints or depth alone.
This synthetic data is designed to be complementary to traditional CG data. While CG provides structural accuracy, diffusion models provide the "texture" of reality, including authentic clothing folds, realistic lighting, and background diversity. The generated dataset comprises 0.79M images with corresponding 3D annotations, covering versatile viewpoints, scenes, and human identities.
Key Results and Generalization
The HumanWild dataset was evaluated across a wide range of benchmarks to verify its effectiveness:
- Benchmark Superiority: On the 3DPW dataset, adding HumanWild data to existing synthetic datasets (BEDLAM + AGORA) improved the Procrustes-Aligned Mean Per-Joint Position Error (PA-MPJPE) from 47.4mm to 44.9mm.
- Handling Perspective Distortion: On the SPEC-MTP benchmark, which features wide-angle lenses and significant distortion, HumanWild outperformed CG-only datasets by a wide margin (90.0mm vs 102.4mm), proving its superior generalization to varied camera setups.
- Complex Poses: The model showed significant gains on the MOYO dataset, which contains challenging yoga poses rarely found in standard mocap datasets.
Conclusion
HumanWild demonstrates that generative models can effectively bridge the gap between synthetic and real-world data for 3D human reconstruction. By eliminating the need for expensive real-world data collection and providing a scalable way to generate high-quality 3D annotations, this approach sets a new standard for HPS training. Future work will focus on improving multi-human scaling and extending the pipeline to human-object interactions and animal pose estimation.
Summary updated