SPG-VTON: Semantic Prediction Guidance for Multi-pose Virtual Try-on

Bingwen Hu, Ping Liu, Zhedong Zheng, Mingwu Ren

Abstract

Image-based virtual try-on is challenging in fitting a target in-shop clothes into a reference person under diverse human poses. Previous works focus on preserving clothing details (e.g., texture, logos, patterns) when transferring desired clothes onto a target person under a fixed pose. However, the performances of existing methods significantly dropped when extending existing methods to multi-pose virtual try-on. In this paper, we propose an end-to-end Semantic Prediction Guidance multi-pose Virtual Try-On Network (SPG-VTON), which could fit the desired clothing into a reference person under arbitrary poses. Concretely, SPG-VTON is composed of three sub-modules: a Semantic Prediction Module (SPM), a Clothes Warping Module (CWM), and a Try-on Synthesis Module (TSM). We evaluate the proposed method on the most massive multi-pose dataset (MPV) and the DeepFashion dataset. The qualitative and quantitative experiments show that SPG-VTON is superior to the state-of-the-art methods and is robust to data noise, showing good scalability to the real-world scenario.

Introduction

Most image-based virtual try-on systems, such as VITON or CP-VTON, are optimized for "fixed pose" scenarios where a person stands straight and faces forward. When these models are applied to multi-pose situations—where subjects may be turning, sitting, or gesturing—the results often suffer from severe distortion, blurred textures, and loss of facial identity.

Previous attempts to address multi-pose try-on, like MG-VTON, utilized multi-stage pipelines that were not end-to-end. This lack of integration often leads to error accumulation; a small mistake in body shape prediction in the early stages can result in a catastrophic failure in the final rendering. Furthermore, many existing models neglect non-body parts like the face, leading to unrealistic results that fail to maintain the subject's identity.

The SPG-VTON Framework

To overcome these limitations, we propose SPG-VTON, an end-to-end network that integrates three specialized sub-modules:

1. Semantic Prediction Module (SPM)

Instead of directly warping clothing onto a raw image, the SPM first generates a predicted semantic map (a pixel-level label for skin, hair, and clothing) for the target pose. This map serves as a spatial "blueprint," guiding the network on exactly where the new garment should be placed relative to the body.

Virtual try-on result for a complex back-view pose, demonstrating the model's ability to handle arbitrary orientations.

2. Clothes Warping Module (CWM)

The CWM transforms flat "in-shop" clothing images to fit the target pose using Thin-Plate Spline (TPS) transformation. A key innovation here is the conductible cycle consistency loss, which ensures that the warping process is reversible. This constraint prevents the texture from becoming distorted or unrecognizable during the transformation.

3. Try-on Synthesis Module (TSM)

The final stage blends the warped clothing with a coarse image of the person. To ensure the final output remains photo-realistic and recognizable, we introduce a face identity loss (utilizing a pre-trained SphereFaceNet) alongside global and local adversarial losses. This keeps the person's facial features sharp and consistent with the original reference image.

Experimental Evaluation

We evaluated SPG-VTON on two major benchmarks: the MPV (Multi-Pose Virtual Try-on) dataset, which contains over 35,000 images and is known for its real-world "noise" (e.g., handbags and hats), and the DeepFashion dataset.

Quantitative Results

SPG-VTON outperformed the previous state-of-the-art (MG-VTON) across key metrics:

  • SSIM (Structural Similarity): Achieved 0.752 on the MPV dataset (vs. 0.744 for MG-VTON).
  • IS (Inception Score): Reached 3.243, indicating superior image realism and variety.

Qualitative Results

Visual assessments confirm that SPG-VTON preserves fine details—such as logos, text, and complex patterns—much more effectively than previous methods, particularly in side-view or occluded poses.

Example of detail preservation where the model successfully maintains the clarity of the graphic logo on the t-shirt.

Conclusion and Insights

The success of SPG-VTON highlights several key insights for the field of computer vision:

  • End-to-End Synergy: Training modules together allows the network to self-correct errors across stages, leading to higher overall quality.
  • Semantics as a Prior: Using human parsing maps provides a stronger geometric constraint than RGB pixels alone, forcing the network to respect human anatomy.
  • Identity Preservation: Specific losses for facial features are crucial for avoiding the "uncanny valley" effect in virtual try-on tasks.

While SPG-VTON represents a significant advancement, future work will focus on closing the gap between predicted and ground-truth semantic maps and extending the architecture to handle extreme occlusions and natural language-based clothing generation.

Summary updated

Abstract

Image-based virtual try-on is challenging in fitting a target in-shop clothes into a reference person under diverse human poses. Previous works focus on preserving clothing details ( e.g., texture, logos, patterns ) when transferring desired clothes onto a target person under a fixed pose. However, the performances of existing methods significantly dropped when extending existing methods to multi-pose virtual try-on. In this paper, we propose an end-to-end Semantic Prediction Guidance multi-pose Virtual Try-On Network (SPG-VTON), which could fit the desired clothing into a reference person under arbitrary poses. Concretely, SPG-VTON is composed of three sub-modules. First, a Semantic Prediction Module (SPM) generates the desired semantic map. The predicted semantic map provides more abundant guidance to locate the desired clothes region and produce a coarse try-on image. Second, a Clothes Warping Module (CWM) warps in-shop clothes to the desired shape according to the predicted semantic map and the desired pose. Specifically, we introduce a conductible cycle consistency loss to alleviate the misalignment in the clothes warping process. Third, a Try-on Synthesis Module (TSM) combines the coarse result and the warped clothes to generate the final virtual try-on image, preserving details of the desired clothes and under the desired pose. Besides, we introduce a face identity loss to refine the facial appearance and maintain the identity of the final virtual try-on result at the same time. We evaluate the proposed method on the most massive multi-pose dataset (MPV) and the DeepFashion dataset. The qualitative and quantitative experiments show that SPG-VTON is superior to the state-of-the-art methods and is robust to the data noise, including background and accessory changes, i.e., hats and handbags, showing good scalability to the real-world scenario.
BibTeX
@article{hu2022spg,
  author = {Bingwen Hu and Ping Liu and Zhedong Zheng and Mingwu Ren},
  title = {{SPG-VTON: Semantic Prediction Guidance for Multi-pose Virtual Try-on}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=N5Ea91s5AKC},
}
Figure