Introduction
Synthesizing realistic human figures has seen significant progress with 3D Generative Adversarial Networks (GANs). However, the task of editing these figures with high precision remains a formidable challenge. Existing approaches often suffer from a lack of localized control—where a global text prompt like "a person in a red shirt" might inadvertently change the color of the pants—and multi-view inconsistency, where edits do not persist correctly across different camera angles. 3DHumanEdit addresses these issues by integrating multi-modal conditioning and a dedicated geometry estimator to ensure localized, stable, and semantically accurate 3D human manipulation.
Methodology
The 3DHumanEdit architecture is designed to decouple semantics and maintain geometric integrity through three core technical pillars.
Multi-modal Feature Fusion
The system processes two primary inputs: a detailed text description and a corresponding segmentation map. Unlike many models that rely solely on CLIP for text, 3DHumanEdit utilizes a T5 encoder. T5's architecture is better suited for processing long, complex descriptions required for full-body outfit manipulation. The segmentation map is processed via a CLIP image encoder. These features are then fused using cross-attention mechanisms to generate "shape-correlated" and "token-correlated" features, ensuring the model understands the relationship between specific words and spatial regions.
Body Part-aware Alignment
To prevent semantic "bleeding," the framework implements a body part-aware alignment loss. This mechanism forces the model's attention to align precisely with pixel-level masks. For instance, the attention map for the "trousers" token is constrained to the corresponding region in the segmentation map, ensuring that edits to the pants do not affect the shirt or shoes.
3D-Aware Manipulation (PSE)
A critical component is the Pose and Shape Estimator (PSE). Rather than requiring manual adjustment of complex Skinned Multi-Person Linear (SMPL) model parameters, the PSE automatically predicts the 3D pose ($\theta$) and shape ($\phi$) directly from the input segmentation. This "geometry-first" approach allows the model to handle the non-rigid nature of the human body more effectively than latent-space-only methods. During training, the model is further regularized using LPIPS and pixel-wise L2 losses to ensure appearance stability across all views.
Experiments and Results
The researchers trained the model using a pipeline involving the DeepFashion dataset and 100,000 synthetic human images. Ground-truth captions were generated using InstructBLIP to provide high-quality, part-specific labels for training.
3DHumanEdit demonstrates superior performance across several benchmarks:
- Image Quality: It achieved a Frechet Inception Distance (FID) of 11.46, significantly outperforming the next best competitor, Text2human (13.81).
- Semantic Accuracy: The model reached a CLIP-Score of 31.30 (ViT-B/32), indicating a higher correlation between the generated image and the textual prompt compared to state-of-the-art baselines.
- Visual Fidelity: Qualitative evaluations show that the framework avoids common artifacts like the "siamese phenomenon" (where a character appears to have two faces when rotated) and maintains sharp, consistent textures across 360-degree views.
Conclusion
3DHumanEdit represents a significant advancement in 3D human avatar manipulation. By decoupling semantic categories and integrating a robust pose/shape estimator, the framework achieves fine-grained control that was previously difficult to attain. While the system remains dependent on the quality of the underlying base GAN, its ability to maintain multi-view consistency and semantic localization sets a new standard for the field. Future work will likely explore more complex poses and the integration of diverse motion sequences.
Summary updated