TL;DR
This paper introduces HumanWild, a novel pipeline for generating large-scale, in-the-wild 3D human pose and shape (HPS) datasets using diffusion models. By combining a customized ControlNet with 3D parametric models and sophisticated denoising techniques, HumanWild produces realistic human images with accurate 3D annotations, significantly improving HPS model generalization to diverse real-world scenarios.
What Problem Does This Paper Address?
Estimating 3D human pose and shape from images is a crucial task with applications in robotics, graphics, and more. However, current state-of-the-art methods heavily rely on datasets that are either confined to indoor motion capture (mocap) studios or generated using computer graphics (CG) rendering. Both approaches have limitations: mocap is expensive and lacks real-world diversity, while CG rendering often struggles with photorealism and creating authentic in-the-wild scenes. This gap in diverse, high-quality, in-the-wild data hinders the generalization capabilities of HPS models to the messy, unpredictable environments encountered in everyday life.
Methods
The core of HumanWild is a diffusion model-based pipeline for generating synthetic human data. Here's a breakdown of the key steps:
- Human-Centric Dataset Curation: They start by collecting a large-scale dataset with comprehensive annotations, including text captions, depth maps, and surface normal images.
- Customized ControlNet: A multi-condition ControlNet is trained on this curated dataset. This model is conditioned on precise 2D keypoints, depth maps, and surface normal images, which are derived from rendering a 3D parametric model (SMPL-X) with specific camera parameters. This allows for fine-grained control over the generated human's pose, shape, and appearance.
- Label Denoising: To address noise in the initial generated labels, they employ a two-step process:
- Segmentation-based Filtering: An off-the-shelf segmentation model (SAM) is used to filter out negative data samples by comparing predicted human masks with ground truth.
- 2D Vertex Estimation: A 2D vertex estimator and the SMPLify algorithm are used to refine the SMPL-X parameters, improving the alignment between the generated image and its 3D annotation.
- Flexible Generation: The pipeline is designed to be flexible, allowing for the generation of diverse human identities, interactions, and in-the-wild scenes, even incorporating wide-angle lens effects.
Datasets
- Training Data for ControlNet:
- Real-world datasets: LAION-Human, COYO-700, PexHuman, and Laion5B (filtered). These provide diverse human images and are annotated with depth, surface normals, and keypoints using pre-trained models.
- CG-rendered dataset: RenderPeople (human models) and AMASS (motion capture sequences) are used to create synthetic data with precise annotations.
- Evaluation Datasets: The generated HumanWild dataset is evaluated on various HPS benchmarks, including 3DPW, AGORA, RICH, and MOYO.
Key Insights and Ideas
- Synthetic data complements CG: The paper strongly suggests that synthetic data generated by diffusion models is not just an alternative but a valuable complement to traditional CG-rendered datasets, especially for achieving better generalization.
- Precise 3D priors are crucial: Conditioning diffusion models on detailed 3D information (keypoints, depth, surface normals derived from SMPL-X) is key to generating accurate and well-aligned human images and annotations.
- Denoising is essential: The presence of noise in initial synthetic data is a significant challenge. The proposed multi-stage denoising process (segmentation filtering and SMPLify refinement) is critical for producing high-quality, usable datasets.
- In-the-wild diversity matters: The authors highlight that the lack of diverse, in-the-wild scenes in existing datasets is a major bottleneck for HPS model generalization. HumanWild directly addresses this by generating such data.
Key Results
- Improved Generalization: HPS models trained with HumanWild data show significantly improved generalization performance on diverse real-world benchmarks, outperforming models trained solely on CG-rendered or limited real-world datasets. For instance, CLIFF trained with HumanWild achieved better results on SPEC-MTP and MOYO.
- Cost-Effective and Scalable: The HumanWild pipeline is presented as being significantly cheaper and more scalable than traditional mocap or CG-based data generation methods.
- High-Quality Annotations: The generated data includes precise 3D annotations (SMPL-X parameters, depth, surface normals) that are well-aligned with the synthetic images.
- Ablation Studies: Experiments demonstrate that using all conditions (keypoint, depth, and surface normal) for ControlNet yields the best image quality and alignment accuracy.
Limitations and Future Work
- Photorealism: The authors acknowledge that their current diffusion model (SDXL-based) can sometimes produce images that are blurry or lack full photorealism, especially in backgrounds. They suggest exploring newer models like FLUX for future improvements.
- Multi-Human Synthesis: The current pipeline is effective for generating images with one to three human subjects. Generating scenes with a larger number of individuals poses challenges due to scale decreases and alignment issues.
- Hand and Face Accuracy: While HumanWild performs well on body metrics, it shows relatively poorer performance on hand and face metrics when evaluated on datasets like AGORA. This is attributed to potential noise in hand/face annotations and limitations in diffusion models' ability to generate fine details at small resolutions.
- Future Directions: The authors envision extending their pipeline to other tasks like 3D animal pose estimation and human-object/human-human interaction reconstruction. They also plan to explore more advanced generative models for greater photorealism.
Summary updated