Abstract
Virtual try-on is a critical technology for online apparel shopping, allowing consumers to fit garments virtually. However, existing image-based methods often suffer from garment shape distortion and loss of texture details. We propose KGI (Keypoints Guided Inpainting), a novel framework that uses graph-based keypoint prediction to guide a semantic-conditioned diffusion model. By extracting human pose and garment keypoints as graphs, the model predicts target garment positions to guide segmentation and warping. Extensive experiments on the VITON-HD dataset demonstrate that KGI produces high-fidelity images that accurately preserve garment shapes and intricate patterns. Code is available at: https://github.com/lizhi-ntu/KGI.
Introduction
Image-based virtual try-on (VTO) aims to seamlessly overlay a garment onto a person's image. Despite significant progress, current state-of-the-art methods face two primary challenges:
- Structural Distortion: Traditional warping techniques like Thin Plate Spline (TPS) often fail during complex poses (e.g., crossed arms), leading to unrealistic stretching or "melting" of the fabric.
- Texture Loss: Blending and GAN-based synthesis frequently blur fine details, such as text logos or intricate floral patterns, making the garment appear "pasted on."
KGI addresses these issues by providing structural guidance through garment "skeletons" and leveraging the high-frequency detail generation of diffusion models.
Methodology
The KGI framework operates in three distinct stages to ensure structural integrity and texture fidelity.
1. Pose-Oriented Garment Keypoints Prediction
The system extracts human pose keypoints and garment landmarks (e.g., necklines, cuffs) and represents them as graphs. A two-stream Graph Convolutional Network (GCN) predicts the target garment keypoints based on the person's pose. This graph-based representation allows the model to understand the "topology" of the clothing, ensuring that structural elements like sleeves remain coherent even under significant deformation.
2. Fine-Grained Warping and Segmentation
To overcome the limitations of global TPS warping, KGI employs a "divide and conquer" strategy. The garment is divided into five sub-segments: left-up, left-low, center, right-up, and right-low. Each segment is warped individually based on the predicted keypoints and then recombined. This approach effectively handles self-occlusions and non-continuous surfaces created by overlapping limbs.
3. Semantic-Conditioned Inpainting
The final try-on image is generated using a Denoising Diffusion Probabilistic Model (DDPM). The model is conditioned on the warped garment, the target segmentation map, and the person's image. Unlike GANs, which may struggle with high-frequency details, the diffusion process "reconstructs" the garment's texture, preserving sharp logos and complex patterns.
Experimental Evaluation
KGI was evaluated on the VITON-HD dataset (13,679 pairs) at resolutions up to 1024x768 pixels.
Quantitative Results
KGI significantly outperformed baselines such as HR-VITON and VITON-HD. At the highest resolution (1024x768), KGI achieved an SSIM of 0.900 and a FID of 6.93, compared to HR-VITON’s 0.892 and 10.91, respectively.
Qualitative Results
Visual comparisons highlight KGI's ability to preserve sharp text (e.g., "Lee" or "VANS" logos) and complex patterns that typically become blurry in other models. It also handles the boundaries between the garment and the person's skin with higher precision.
Conclusion
The KGI framework demonstrates that graph-based structural guidance combined with diffusion-based inpainting can effectively solve garment distortion and texture loss in virtual try-on. While currently focused on upper-body apparel, the methodology provides a robust foundation for full-body virtual styling in the future.
Summary updated