TL;DR
ClotheDreamer tackles the challenge of generating high-quality, production-ready 3D garments from text descriptions. It introduces a novel "Disentangled Clothe Gaussian Splatting" (DCGS) representation that separates garment geometry from the human body, enabling easier reuse and animation. This method achieves superior text consistency and garment quality compared to existing approaches.
What Problem Does This Paper Address?
Creating realistic 3D garments from text prompts is a significant challenge in digital avatar creation. Existing methods often tightly couple the garment with the human body, making it difficult to edit, reuse, or animate the garment independently. This limitation hinders applications like virtual try-on and realistic character animation. Furthermore, generating production-ready assets with fine geometric details and realistic material properties from text alone remains an open problem.
Methods
ClotheDreamer introduces a novel Disentangled Clothe Gaussian Splatting (DCGS) representation. Instead of treating the clothed avatar as a single entity, DCGS initializes the entire avatar as a single 3D Gaussian model but then freezes the Gaussians representing the human body. This allows for the separate optimization of the garment Gaussians.
Key techniques include:
- ID-Based DCGS Initialization: Leveraging the body segmentation from SMPL-X models, ClotheDreamer categorizes clothing types (e.g., "no sleeve & shorts," "long sleeve & pants") to initialize the garment Gaussians. This provides a stronger prior than generic point clouds.
- Bidirectional SDS Guidance: To ensure high-quality and complete garment generation, the method uses Score Distillation Sampling (SDS) guidance. It applies this guidance bidirectionally to supervise both the clothed avatar's and the garment's RGB and depth renderings, conditioned on pose information. This helps regularize the underlying geometry.
- Loose Clothing Pruning Strategy: A new pruning strategy is introduced to specifically handle loose clothing (like gowns or dresses). This strategy aims to avoid mistakenly removing useful Gaussian splats, which can happen with existing methods, thus preserving the garment's completeness.
- Template-Guided Generation: The framework supports custom clothing templates (e.g., meshes from Marvelous Designer) to guide the generation process, allowing for personalized garment creation.
- Animation: The disentangled nature of the garments allows for animation using simulated mesh sequences, overcoming limitations of directly binding garments to SMPL-X for animation, especially for loose clothing.
Datasets
The paper does not explicitly mention training on a specific large-scale 3D garment dataset. Instead, it leverages pre-trained 2D diffusion models (like Stable Diffusion) for guidance via Score Distillation Sampling (SDS). For initialization and evaluation, it utilizes SMPL-X body models and their associated segmentation information. The paper also mentions using templates from Marvelous Designer for guided generation.
Key Insights and Ideas
- Decoupling is Key: The core insight is that disentangling the garment from the human body is crucial for creating reusable and animatable 3D assets. This separation is achieved through the DCGS representation and selective optimization.
- Body as a Frozen Prior: Freezing the body Gaussians while optimizing the garment Gaussians is an effective way to leverage existing body models without letting them interfere with garment-specific details.
- Bidirectional Guidance for Robustness: Applying SDS guidance to both the full avatar and the garment separately, and incorporating depth information, leads to more robust and complete geometry, especially for complex or loose clothing.
- Pruning for Loose Clothing: The specific pruning strategy for loose clothing highlights a nuanced understanding of the challenges in Gaussian-based generation, where aggressive pruning can be detrimental.
- 3D Gaussians for Garments: The paper demonstrates the suitability of 3D Gaussian Splatting for generating detailed and realistic garment geometry and textures, offering advantages in rendering speed and editability.
Key Results
- Superior Text Consistency: ClotheDreamer achieved the highest CLIP score (31.464) among compared methods, indicating superior semantic similarity between the generated garments and the text prompts.
- Highest User Preference: A user study showed that ClotheDreamer received the highest preference score (4.55) from participants, indicating better overall quality and realism.
- High-Fidelity and Wearable Assets: The method generates diverse, wearable, and production-ready 3D garment assets with refined geometry and realistic appearance details.
- Effective Animation: The generated garments can be animated realistically, even for loose clothing, by leveraging simulated mesh sequences.
- Template-Guided Personalization: The approach successfully incorporates custom template meshes for personalized garment generation.
Limitations and Future Work
- Integrated Upper/Lower Garments: Currently, the method integrates upper and lower garments. More refined decoupling for complex try-on scenarios is a potential area for improvement.
- Color Oversaturation: Similar to other SDS-based methods, ClotheDreamer can sometimes suffer from color oversaturation. Future work could explore techniques to mitigate this.
- Lighting Disentanglement: Disentangling lighting from 3D Gaussian representations is suggested as an interesting future direction to enhance realism.
- More Complex Try-on Scenarios: The paper suggests that a more refined decoupling of garments would be beneficial for more complex try-on scenarios.
Summary updated