Virtual Try-On with Pose-Garment Keypoints Guided Inpainting

Zhi Li, Pengfei Wei, Xiang Yin, Zejun Ma, Alex C. KotICCV 2023 · OpenReview

Abstract

Virtual try-on is a critical technology for online apparel shopping, allowing consumers to fit garments virtually. However, existing image-based methods often suffer from garment shape distortion and loss of texture details. We propose KGI (Keypoints Guided Inpainting), a novel framework that uses graph-based keypoint prediction to guide a semantic-conditioned diffusion model. By extracting human pose and garment keypoints as graphs, the model predicts target garment positions to guide segmentation and warping. Extensive experiments on the VITON-HD dataset demonstrate that KGI produces high-fidelity images that accurately preserve garment shapes and intricate patterns. Code is available at: https://github.com/lizhi-ntu/KGI.

Introduction

Image-based virtual try-on (VTO) aims to seamlessly overlay a garment onto a person's image. Despite significant progress, current state-of-the-art methods face two primary challenges:

  1. Structural Distortion: Traditional warping techniques like Thin Plate Spline (TPS) often fail during complex poses (e.g., crossed arms), leading to unrealistic stretching or "melting" of the fabric.
  2. Texture Loss: Blending and GAN-based synthesis frequently blur fine details, such as text logos or intricate floral patterns, making the garment appear "pasted on."

KGI addresses these issues by providing structural guidance through garment "skeletons" and leveraging the high-frequency detail generation of diffusion models.

Methodology

The KGI framework operates in three distinct stages to ensure structural integrity and texture fidelity.

Overview of the KGI framework showing the three-stage pipeline from keypoint prediction to final inpainting.

1. Pose-Oriented Garment Keypoints Prediction

The system extracts human pose keypoints and garment landmarks (e.g., necklines, cuffs) and represents them as graphs. A two-stream Graph Convolutional Network (GCN) predicts the target garment keypoints based on the person's pose. This graph-based representation allows the model to understand the "topology" of the clothing, ensuring that structural elements like sleeves remain coherent even under significant deformation.

2. Fine-Grained Warping and Segmentation

To overcome the limitations of global TPS warping, KGI employs a "divide and conquer" strategy. The garment is divided into five sub-segments: left-up, left-low, center, right-up, and right-low. Each segment is warped individually based on the predicted keypoints and then recombined. This approach effectively handles self-occlusions and non-continuous surfaces created by overlapping limbs.

Comparison between conventional TPS warping and the proposed 5-segment fine-grained warping.

3. Semantic-Conditioned Inpainting

The final try-on image is generated using a Denoising Diffusion Probabilistic Model (DDPM). The model is conditioned on the warped garment, the target segmentation map, and the person's image. Unlike GANs, which may struggle with high-frequency details, the diffusion process "reconstructs" the garment's texture, preserving sharp logos and complex patterns.

Experimental Evaluation

KGI was evaluated on the VITON-HD dataset (13,679 pairs) at resolutions up to 1024x768 pixels.

Quantitative Results

KGI significantly outperformed baselines such as HR-VITON and VITON-HD. At the highest resolution (1024x768), KGI achieved an SSIM of 0.900 and a FID of 6.93, compared to HR-VITON’s 0.892 and 10.91, respectively.

Qualitative Results

Visual comparisons highlight KGI's ability to preserve sharp text (e.g., "Lee" or "VANS" logos) and complex patterns that typically become blurry in other models. It also handles the boundaries between the garment and the person's skin with higher precision.

Conclusion

The KGI framework demonstrates that graph-based structural guidance combined with diffusion-based inpainting can effectively solve garment distortion and texture loss in virtual try-on. While currently focused on upper-body apparel, the methodology provides a robust foundation for full-body virtual styling in the future.

Summary updated

Abstract

Virtual try-on is an important technology supporting on-line apparel shopping, which provides consumers with a virtual experience to fit garments without physically wearing them. Recently, the image-based virtual try-on has received growing research attention. However, the synthetic results of existing virtual try-on methods usually present distortions in garment shape and lose pattern details. In this paper, we propose a pose-garment keypoints guided inpainting method for the image-based virtual try-on task, which produces high-fidelity try-on images and well preserves the shapes and patterns of the garments. In our method, human pose and garment keypoints are extracted from source images and constructed as graphs to predict the garment keypoints at the target pose. After which, the predicted key-points are used as guide information to predict the target segmentation map and warp the garment image. The try-on image is finally generated with a semantic-conditioned inpainting scheme using the segmentation map and recomposed person image as conditions. To verify the effectiveness of our proposed method, we conduct extensive experiments on the VITON-HD dataset under both paired and unpaired experimental settings. The qualitative and quantitative results show that our method significantly outperforms prior methods at different image resolutions. The codes repository link is https://github.com/lizhi-ntu/KGI.
BibTeX
@inproceedings{li2023virtual,
  author = {Zhi Li and Pengfei Wei and Xiang Yin and Zejun Ma and Alex C. Kot},
  title = {{Virtual Try-On with Pose-Garment Keypoints Guided Inpainting}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=wvDoKXNxam},
}
Figure