Introduction
Image-based Virtual Try-On (VTON) has advanced significantly with the advent of Latent Diffusion Models (LDMs). However, a persistent "Achilles' heel" remains: hand occlusion. In many real-world fashion images, models do not simply stand with arms at their sides; they cross their arms, put hands in pockets, or touch the garments. Existing VTON models often fail in these scenarios, producing "melted" fingers or anatomically impossible artifacts. VTON-HandFit addresses this by disentangling hand structure from appearance and utilizing 3D hand priors to guide the generation process.
The Challenge of Hand Occlusion
Traditional VTON methods often rely on "parsing"—cutting and pasting the original hand onto the new garment—which leaves visible artifacts and background remnants. Alternatively, generative models attempting to reconstruct hands from scratch frequently struggle with the complexity and flexibility of human hands.
As seen in the figure above, complex poses like crossed arms present a significant challenge for maintaining both garment integrity and anatomical correctness. VTON-HandFit treats this as a guided synthesis task, ensuring that the hand's 3D structure is respected during the diffusion process.
Methodology: Disentangling Structure and Appearance
The core innovation of VTON-HandFit is the Hand-feature Disentanglement Embedding, which separates hand information into two distinct streams:
- Structure-parametric features: Utilizing the HaMeR model, the system extracts 3D MANO parameters (joint locations and rotations). This provides a mathematical blueprint of the hand's 3D shape, which is far more robust than 2D skeletons for preventing "flat" or "twisted" artifacts.
- Visual-appearance features: DINOv2 is used to extract high-level texture and color information from cropped hand images, ensuring the generated hand matches the model's original skin tone and texture.
To prevent these hand features from distorting the garment, the authors implemented Masked Cross-Attention. This ensures that hand-specific priors only influence the pixels designated as "hand," while a Hand-Canny Constraint Loss forces the model to respect sharp boundaries and fine finger details during training.
Results and the Handfit-3K Dataset
The researchers evaluated VTON-HandFit on standard benchmarks (VITON-HD, DressCode) and a new specialized dataset, Handfit-3K. This new dataset consists of 3,620 high-quality test cases specifically featuring complex hand occlusions to stress-test model performance.
Qualitative results show that VTON-HandFit maintains clean edges and realistic skin-to-fabric transitions, even when accessories like rings are present. Quantitatively, the model achieved a lower FID (8.8178) on VITON-HD and significantly reduced the Mean Per Joint Position Error (MPJPE) to 25.13 on the Handfit-3K dataset, outperforming state-of-the-art baselines like IDM-VTON and OOTDiffusion.
Conclusion
VTON-HandFit represents a significant step forward in making virtual try-on technology viable for diverse, real-world fashion photography. By integrating 3D hand priors and disentangled feature embeddings, the framework effectively solves the "melted hand" problem, producing results that are both visually pleasing and anatomically accurate. Future work may look to extend these occlusion-handling techniques to other complex elements like hair, jewelry, or handheld accessories.
Summary updated