TL;DR
This paper introduces HumanCoser, a novel framework for generating high-fidelity, layered 3D humans from text prompts. It achieves physically decoupled generation of body and clothing, enabling independent editing and seamless transfer of garments between different avatars, a significant step beyond existing coupled generation methods.
What Problem Does This Paper Address?
Generating realistic 3D human models, especially when clothed, is a challenging task. Existing methods often produce a single, inseparable mesh for the body and clothing. This "coupled" representation severely limits editing capabilities; you can't easily change a person's body shape without affecting the clothing, or vice-versa, nor can you transfer a garment from one avatar to another with a different body type. This lack of modularity hinders applications in areas like virtual try-on, gaming, and animation where flexible editing and reuse are crucial. The paper aims to solve this by creating a layered 3D human generation system that disentangles body and clothing while maintaining structural consistency and semantic alignment with text descriptions.
Methods
HumanCoser employs a two-stage approach:
- Decoupled Generation Stage:
- Physically Decoupled Diffusion Model: The core innovation is a diffusion model that generates body and clothing as separate, yet structurally consistent, layers. This is achieved using uniform shape priors derived from the SMPL (Skinned Multi-Person Linear Model) for both body and clothing, allowing for more diverse 3D content generation.
- Layered Geometric Prior: To ensure structural consistency, a layered shape prior is used, combined with predicted normals and regularization constraints.
- Canonical Spatial Sampling & Density Initialization: To mitigate body self-occlusion issues and standardize input for diffusion, points are projected into a canonical space, and density fields are initialized using the layered geometric priors.
- Smooth Normals: A normal prediction network is used to generate smoother surface normals, improving coloring and geometric detail.
- Matching and Synthesis Stage:
- 3D Semantic Confidence Network: To ensure generated clothing accurately matches the text prompt, a semantic-aware network and a 3D semantic weight prediction network are introduced. This network identifies and eliminates non-clothing semantic content, enhancing robustness through Gaussian noise.
- SMPL-driven Implicit Field Deformation Network (SID Net): To enable clothing transfer and reuse across different body shapes, this network uses SMPL-X (an extension of SMPL) for body shape priors and learnable vertex offsets. It deforms the body's implicit field to precisely match the clothing's geometry, facilitating seamless garment fitting.
Datasets
The paper does not explicitly mention specific datasets used for training or evaluation. However, it states that the framework is implemented based on Stable Diffusion and NeRF components, implying reliance on general image and 3D scene data that these models are trained on.
Key Insights and Ideas
- Decoupling is Key for Editability: The fundamental insight is that separating body and clothing generation is essential for enabling independent editing and flexible garment transfer.
- Semantic Confidence for Accurate Clothing: Using a semantic confidence strategy is crucial for ensuring that the generated clothing truly reflects the text prompt, filtering out unwanted artifacts.
- SMPL as a Universal Fit Tool: Leveraging SMPL-X for body representation and its vertex prediction capabilities allows for a robust mechanism to adapt clothing to diverse body shapes.
- Layered Priors for Consistency: The use of layered shape priors for both body and clothing is vital for maintaining structural integrity across the generated components.
Key Results
- Qualitative Improvements: The method generates 3D humans with consistent body structures and allows for free, layered editing. Qualitative comparisons show HumanCoser producing more detailed geometry and textures, including loose clothing and long hair, compared to state-of-the-art coupled generation methods.
- Clothing Transfer and Reuse: The paper demonstrates successful transfer of clothing between avatars with different body shapes, showcasing the practical utility of the decoupled approach.
- Quantitative Evaluation: A user study indicated that HumanCoser achieved superior scores in geometric quality, texture quality, and consistency with text prompts compared to AvatarCLIP, TADA, and Latent-NeRF.
Limitations and Future Work
- Unnatural Matching: The authors acknowledge that while their method fits bodies to clothing of various shapes, it may result in unnatural matching outcomes when body and clothing shapes differ significantly.
- No Uniform Clothing Template: The absence of a universal parametric clothing template prevents direct differentiable rendering for matching loss optimization using a uniform 3D proxy.
- Future Work: The authors suggest employing more accurate deformation proxies and integrating object collision detection to improve the bidirectional matching of clothing and body for even better layered generation quality.
Summary updated