Introduction
Personalized image generation has traditionally struggled with two significant hurdles: capturing a person's unique body shape and managing complex interactions between multiple individuals. Most existing models focus heavily on facial identity, often grafting a user's face onto a generic body template. Furthermore, when scenes involve overlapping people—such as hugging or group activities—standard 2D pose skeletons often fail, leading to "anatomical soup" where limbs merge or distort.
Overcoming the "Anatomical Soup"
PersonaCraft introduces a framework that moves beyond 2D skeletons by utilizing 3D human models (SMPLx) as conditioning. This provides the diffusion model with depth information and surface orientations, which are essential for resolving spatial ambiguities that 2D lines cannot capture.
To specifically address overlapping regions, the authors developed the Occlusion Boundary Enhancer Network (OccNet) and Occlusion-Aware Classifier-Free Guidance (OccCFG). These components act as fine-detail specialists for the boundaries where one person's body crosses in front of another's, significantly reducing visual artifacts in crowded scenes.
Methodology: Beyond Facial Identity
The framework achieves full-body personalization through a dual-pathway approach. It extracts structural shape parameters from a reference photo using the SMPLx model and combines them with textual descriptions of body types (e.g., "muscular" or "slender"). This ensures that the generated character reflects the reference person's actual build and physical presence, not just their face.
Key Results and Efficiency
PersonaCraft demonstrates a significant leap over state-of-the-art baselines like InstantID and IP-Adapter across several metrics:
- Pose Accuracy: It nearly halves the 3D pose error (MPJPE) compared to the best baselines, scoring 60.65 against baseline errors of over 112.
- Identity Preservation: It excels in "Multi-Identity" scenarios, preventing the "bleeding" of features between different individuals in the same scene.
- Efficiency: Unlike optimization-based methods that can take over 10 minutes to learn a new identity, PersonaCraft's inference-based approach takes roughly 17 seconds, making it viable for real-time applications.
Conclusion
By integrating 3D geometry into the 2D diffusion process, PersonaCraft provides a robust solution for generating complex, multi-person scenes with high identity fidelity. While it currently depends on the accuracy of initial 3D fitting, it sets a new standard for personalized full-body synthesis that respects both facial identity and physical build.
Summary updated