VTON-HandFit: Virtual Try-on for Arbitrary Hand Pose Guided by Hand Priors Embedding

Yujie Liang, Xiaobin Hu, Boyuan Jiang, Donghao Luo, Kai Wu, Wenhui Han, Taisong Jin, Chengjie Wang

Introduction

Image-based Virtual Try-On (VTON) has advanced significantly with the advent of Latent Diffusion Models (LDMs). However, a persistent "Achilles' heel" remains: hand occlusion. In many real-world fashion images, models do not simply stand with arms at their sides; they cross their arms, put hands in pockets, or touch the garments. Existing VTON models often fail in these scenarios, producing "melted" fingers or anatomically impossible artifacts. VTON-HandFit addresses this by disentangling hand structure from appearance and utilizing 3D hand priors to guide the generation process.

The Challenge of Hand Occlusion

Traditional VTON methods often rely on "parsing"—cutting and pasting the original hand onto the new garment—which leaves visible artifacts and background remnants. Alternatively, generative models attempting to reconstruct hands from scratch frequently struggle with the complexity and flexibility of human hands.

VTON-HandFit successfully reconstructs complex poses like crossed arms, avoiding the 'melted' artifacts common in previous models.

As seen in the figure above, complex poses like crossed arms present a significant challenge for maintaining both garment integrity and anatomical correctness. VTON-HandFit treats this as a guided synthesis task, ensuring that the hand's 3D structure is respected during the diffusion process.

Methodology: Disentangling Structure and Appearance

The core innovation of VTON-HandFit is the Hand-feature Disentanglement Embedding, which separates hand information into two distinct streams:

  1. Structure-parametric features: Utilizing the HaMeR model, the system extracts 3D MANO parameters (joint locations and rotations). This provides a mathematical blueprint of the hand's 3D shape, which is far more robust than 2D skeletons for preventing "flat" or "twisted" artifacts.
  2. Visual-appearance features: DINOv2 is used to extract high-level texture and color information from cropped hand images, ensuring the generated hand matches the model's original skin tone and texture.

To prevent these hand features from distorting the garment, the authors implemented Masked Cross-Attention. This ensures that hand-specific priors only influence the pixels designated as "hand," while a Hand-Canny Constraint Loss forces the model to respect sharp boundaries and fine finger details during training.

Results and the Handfit-3K Dataset

The researchers evaluated VTON-HandFit on standard benchmarks (VITON-HD, DressCode) and a new specialized dataset, Handfit-3K. This new dataset consists of 3,620 high-quality test cases specifically featuring complex hand occlusions to stress-test model performance.

The model maintains sharp boundaries and realistic textures, even for fine details like rings and skin-to-fabric transitions.

Qualitative results show that VTON-HandFit maintains clean edges and realistic skin-to-fabric transitions, even when accessories like rings are present. Quantitatively, the model achieved a lower FID (8.8178) on VITON-HD and significantly reduced the Mean Per Joint Position Error (MPJPE) to 25.13 on the Handfit-3K dataset, outperforming state-of-the-art baselines like IDM-VTON and OOTDiffusion.

Conclusion

VTON-HandFit represents a significant step forward in making virtual try-on technology viable for diverse, real-world fashion photography. By integrating 3D hand priors and disentangled feature embeddings, the framework effectively solves the "melted hand" problem, producing results that are both visually pleasing and anatomically accurate. Future work may look to extend these occlusion-handling techniques to other complex elements like hair, jewelry, or handheld accessories.

Summary updated

BibTeX
@inproceedings{liang2025vton,
  author = {Yujie Liang and Xiaobin Hu and Boyuan Jiang and Donghao Luo and Kai Wu and Wenhui Han and Taisong Jin and Chengjie Wang},
  title = {{VTON-HandFit: Virtual Try-on for Arbitrary Hand Pose Guided by Hand Priors Embedding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02106},
}
Figure