Layered 3D Human Generation via Semantic-Aware Diffusion Model

Yi Wang, Jian Ma, Ruizhi Shao, Qiao Feng, Yu‐Kun Lai, Yebin Liu, Kun Li

TL;DR

This paper introduces HumanCoser, a novel framework for generating high-fidelity, layered 3D humans from text prompts. It achieves physically decoupled generation of body and clothing, enabling independent editing and seamless transfer of garments between different avatars, a significant step beyond existing coupled generation methods.

What Problem Does This Paper Address?

Generating realistic 3D human models, especially when clothed, is a challenging task. Existing methods often produce a single, inseparable mesh for the body and clothing. This "coupled" representation severely limits editing capabilities; you can't easily change a person's body shape without affecting the clothing, or vice-versa, nor can you transfer a garment from one avatar to another with a different body type. This lack of modularity hinders applications in areas like virtual try-on, gaming, and animation where flexible editing and reuse are crucial. The paper aims to solve this by creating a layered 3D human generation system that disentangles body and clothing while maintaining structural consistency and semantic alignment with text descriptions.

Methods

HumanCoser employs a two-stage approach:

  1. Decoupled Generation Stage:
    • Physically Decoupled Diffusion Model: The core innovation is a diffusion model that generates body and clothing as separate, yet structurally consistent, layers. This is achieved using uniform shape priors derived from the SMPL (Skinned Multi-Person Linear Model) for both body and clothing, allowing for more diverse 3D content generation.
  2. Layered Geometric Prior: To ensure structural consistency, a layered shape prior is used, combined with predicted normals and regularization constraints.
  3. Canonical Spatial Sampling & Density Initialization: To mitigate body self-occlusion issues and standardize input for diffusion, points are projected into a canonical space, and density fields are initialized using the layered geometric priors.
  4. Smooth Normals: A normal prediction network is used to generate smoother surface normals, improving coloring and geometric detail.
  5. Matching and Synthesis Stage:
    • 3D Semantic Confidence Network: To ensure generated clothing accurately matches the text prompt, a semantic-aware network and a 3D semantic weight prediction network are introduced. This network identifies and eliminates non-clothing semantic content, enhancing robustness through Gaussian noise.
  6. SMPL-driven Implicit Field Deformation Network (SID Net): To enable clothing transfer and reuse across different body shapes, this network uses SMPL-X (an extension of SMPL) for body shape priors and learnable vertex offsets. It deforms the body's implicit field to precisely match the clothing's geometry, facilitating seamless garment fitting.

Datasets

The paper does not explicitly mention specific datasets used for training or evaluation. However, it states that the framework is implemented based on Stable Diffusion and NeRF components, implying reliance on general image and 3D scene data that these models are trained on.

Key Insights and Ideas

  • Decoupling is Key for Editability: The fundamental insight is that separating body and clothing generation is essential for enabling independent editing and flexible garment transfer.
  • Semantic Confidence for Accurate Clothing: Using a semantic confidence strategy is crucial for ensuring that the generated clothing truly reflects the text prompt, filtering out unwanted artifacts.
  • SMPL as a Universal Fit Tool: Leveraging SMPL-X for body representation and its vertex prediction capabilities allows for a robust mechanism to adapt clothing to diverse body shapes.
  • Layered Priors for Consistency: The use of layered shape priors for both body and clothing is vital for maintaining structural integrity across the generated components.

Key Results

  • Qualitative Improvements: The method generates 3D humans with consistent body structures and allows for free, layered editing. Qualitative comparisons show HumanCoser producing more detailed geometry and textures, including loose clothing and long hair, compared to state-of-the-art coupled generation methods.
  • Clothing Transfer and Reuse: The paper demonstrates successful transfer of clothing between avatars with different body shapes, showcasing the practical utility of the decoupled approach.
  • Quantitative Evaluation: A user study indicated that HumanCoser achieved superior scores in geometric quality, texture quality, and consistency with text prompts compared to AvatarCLIP, TADA, and Latent-NeRF.

Limitations and Future Work

  • Unnatural Matching: The authors acknowledge that while their method fits bodies to clothing of various shapes, it may result in unnatural matching outcomes when body and clothing shapes differ significantly.
  • No Uniform Clothing Template: The absence of a universal parametric clothing template prevents direct differentiable rendering for matching loss optimization using a uniform 3D proxy.
  • Future Work: The authors suggest employing more accurate deformation proxies and integrating object collision detection to improve the bidirectional matching of clothing and body for even better layered generation quality.

Summary updated

Abstract

The generation of 3D clothed humans has attracted increasing attention in recent years. However, existing work cannot generate layered high-quality 3D humans with consistent body structures. As a result, these methods are unable to arbitrarily and separately change and edit the body and clothing of the human. In this paper, we propose a text-driven layered 3D human generation framework based on a novel physically-decoupled semantic-aware diffusion model. To keep the generated clothing consistent with the target text, we propose a semantic-confidence strategy for clothing that can eliminate the non-clothing content generated by the model. To match the clothing with different body shapes, we propose a SMPL-driven implicit field deformation network that enables the free transfer and reuse of clothing. Besides, we introduce uniform shape priors based on the SMPL model for body and clothing, respectively, which generates more diverse 3D content without being constrained by specific templates. The experimental results demonstrate that the proposed method not only generates 3D humans with consistent body structures but also allows free editing in a layered manner. The source code will be made public.
BibTeX
@misc{wang2023layered,
  author = {Yi Wang and Jian Ma and Ruizhi Shao and Qiao Feng and Yu‐Kun Lai and Yebin Liu and Kun Li},
  title = {{Layered 3D Human Generation via Semantic-Aware Diffusion Model}},
  year = {2023},
  eprint = {2312.05804},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.05804},
}
Figure