SPG-VTON: Semantic Prediction Guidance for Multi-Pose Virtual Try-on

Bingwen Hu, Ping Liu, Zhedong Zheng, Mingwu Ren

TL;DR

SPG-VTON is a new end-to-end framework for virtual try-on that successfully fits clothing onto people in arbitrary, complex poses. By using semantic prediction as a "blueprint" and introducing a novel conductible cycle consistency loss, the model generates photo-realistic images that preserve both the fine details of the clothes and the identity of the person.

What Problem Does This Paper Address?

Most image-based virtual try-on systems (like the well-known VITON or CP-VTON) are designed for a "fixed pose"—usually a person standing straight and facing forward. When you try to apply these models to "multi-pose" scenarios—where a person might be turning, sitting, or gesturing—the results often fall apart. The clothes look distorted, textures get blurred, and the person’s face can lose its identity.

Previous attempts to solve this, such as MG-VTON, used multi-stage pipelines. However, these pipelines are not "end-to-end," meaning each step is optimized in isolation. This leads to error accumulation: if the first stage makes a small mistake in predicting the body shape, that error gets magnified in the final rendering. Furthermore, many models ignore "non-body" parts like the face, leading to unrealistic or "uncanny" results.

Methods

The authors propose SPG-VTON (Semantic Prediction Guidance Virtual Try-on Network), which consists of three integrated sub-modules trained in an end-to-end fashion:

  1. Semantic Prediction Module (SPM): Instead of just trying to "paste" clothes, the SPM first predicts a semantic map (a pixel-level label of what is skin, hair, or clothing) for the target pose. This map acts as a spatial guide to tell the network exactly where the new clothes should go.
  2. Clothes Warping Module (CWM): This module takes the flat "in-shop" clothing image and warps it to fit the target pose using Thin-Plate Spline (TPS) transformation. The "secret sauce" here is a conductible cycle consistency loss. This ensures that if you warp the clothes to the person and then try to warp them back to the original flat shape, they match. This prevents the texture from becoming a distorted mess.
  3. Try-on Synthesis Module (TSM): This final stage blends the warped clothes with a coarse image of the person. To keep the person looking like themselves, the authors introduced a face identity loss (using a pre-trained SphereFaceNet) and both global and local adversarial losses to ensure the final image is sharp and realistic.

Datasets

The model was evaluated on two major benchmarks:

  • MPV (Multi-Pose Virtual Try-on): Currently the largest dataset for this task, containing over 35,000 person images and 13,000 clothing images. It is notoriously "noisy," containing real-world artifacts like handbags, hats, and varied backgrounds.
  • DeepFashion: A widely used dataset for clothing recognition and generation. The authors used a subset of 10,000 pairs to test the model’s generalization.

Key Insights and Ideas

  • End-to-End is Essential: By training all modules together, the network can "self-correct." If the semantic prediction is slightly off, the synthesis module can learn to compensate, leading to much higher overall quality than multi-stage pipelines.
  • Semantics as a Prior: Using a human parsing map as a middle-man between the input and output provides a much stronger geometric constraint than just looking at RGB pixels. It forces the network to understand human anatomy before it tries to "dress" the person.
  • Identity Preservation: Most try-on papers focus solely on the clothes. By adding a specific loss for the face, SPG-VTON avoids the common pitfall where the person's facial features become blurred or altered during the clothing transfer.
  • Robustness to "Label Noise": The authors found that their model could handle "noisy" training data (e.g., images where the person is holding a bag that isn't in the target image) much better than previous methods, making it more viable for real-world web-scraped data.

Key Results

SPG-VTON outperformed the previous state-of-the-art (MG-VTON) in both quantitative and qualitative metrics:

  • SSIM (Structural Similarity): Achieved 0.752 on the MPV dataset, compared to MG-VTON’s 0.744.
  • IS (Inception Score): Reached 3.243, indicating higher image variety and realism.
  • Qualitative: Visual results show significantly better preservation of clothing logos and textures, especially in side-view poses or complex arm positions where other models typically fail.

Limitations and Future Work

While SPG-VTON is a significant step forward, the authors note a few areas for improvement:

  • Semantic Gap: There is still a performance gap between using "predicted" semantic maps and "ground-truth" maps. Improving the accuracy of the initial SPM would yield even better results.
  • Beyond Clothing: The authors suggest that this architecture could be adapted for other synthesis tasks, such as vehicle appearance design or generating clothing based on natural language descriptions (e.g., "Put a red floral dress on this person").
  • Extreme Poses: While it handles "arbitrary" poses better than others, extreme occlusions (where a body part is completely hidden) still pose a challenge for texture consistency.

Summary updated

Abstract

Image-based virtual try-on is challenging in fitting a target in-shop clothes onto a reference person under diverse human poses. Previous works focus on preserving clothing details ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.,</i> texture, logos, patterns) when transferring desired clothes onto a target person under a fixed pose. However, the performances of existing methods significantly dropped when extending existing methods to multi-pose virtual try-on. In this paper, we propose an end-to-end Semantic Prediction Guidance multi-pose Virtual Try-On Network (SPG-VTON), which can fit the desired clothing into a reference person under arbitrary poses. Specifically, SPG-VTON is composed of three sub-modules. First, a Semantic Prediction Module (SPM) generates the desired semantic map. The predicted semantic map provides more abundant guidance to locate the desired clothing region and produce a coarse try-on image. Second, a Clothes Warping Module (CWM) warps in-shop clothes to the desired shape according to the predicted semantic map and the desired pose. Specifically, we introduce a conductible cycle consistency loss to alleviate the misalignment in the clothing warping process. Third, a Try-on Synthesis Module (TSM) combines the coarse result and the warped clothes to generate the final virtual try-on image, preserving details of the desired clothes and under the desired pose. In addition, we introduce a face identity loss to refine the facial appearance and maintain the identity of the final virtual try-on result at the same time. We evaluate the proposed method on the most massive multi-pose dataset (MPV) and the DeepFashion dataset. The qualitative and quantitative experiments show that SPG-VTON is superior to the state-of-the-art methods and is robust to data noise, including background and accessory changes, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , hats and handbags, showing good scalability to the real-world scenario.
BibTeX
@article{hu2022spg,
  author = {Bingwen Hu and Ping Liu and Zhedong Zheng and Mingwu Ren},
  title = {{SPG-VTON: Semantic Prediction Guidance for Multi-Pose Virtual Try-on}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=N5Ea91s5AKC},
}

Loading related content...

Figure