Abstract
Image-based virtual try-on is challenging in fitting a target in-shop clothes into a reference person under diverse human poses. Previous works focus on preserving clothing details (e.g., texture, logos, patterns) when transferring desired clothes onto a target person under a fixed pose. However, the performances of existing methods significantly dropped when extending existing methods to multi-pose virtual try-on. In this paper, we propose an end-to-end Semantic Prediction Guidance multi-pose Virtual Try-On Network (SPG-VTON), which could fit the desired clothing into a reference person under arbitrary poses. Concretely, SPG-VTON is composed of three sub-modules: a Semantic Prediction Module (SPM), a Clothes Warping Module (CWM), and a Try-on Synthesis Module (TSM). We evaluate the proposed method on the most massive multi-pose dataset (MPV) and the DeepFashion dataset. The qualitative and quantitative experiments show that SPG-VTON is superior to the state-of-the-art methods and is robust to data noise, showing good scalability to the real-world scenario.
Introduction
Most image-based virtual try-on systems, such as VITON or CP-VTON, are optimized for "fixed pose" scenarios where a person stands straight and faces forward. When these models are applied to multi-pose situations—where subjects may be turning, sitting, or gesturing—the results often suffer from severe distortion, blurred textures, and loss of facial identity.
Previous attempts to address multi-pose try-on, like MG-VTON, utilized multi-stage pipelines that were not end-to-end. This lack of integration often leads to error accumulation; a small mistake in body shape prediction in the early stages can result in a catastrophic failure in the final rendering. Furthermore, many existing models neglect non-body parts like the face, leading to unrealistic results that fail to maintain the subject's identity.
The SPG-VTON Framework
To overcome these limitations, we propose SPG-VTON, an end-to-end network that integrates three specialized sub-modules:
1. Semantic Prediction Module (SPM)
Instead of directly warping clothing onto a raw image, the SPM first generates a predicted semantic map (a pixel-level label for skin, hair, and clothing) for the target pose. This map serves as a spatial "blueprint," guiding the network on exactly where the new garment should be placed relative to the body.
2. Clothes Warping Module (CWM)
The CWM transforms flat "in-shop" clothing images to fit the target pose using Thin-Plate Spline (TPS) transformation. A key innovation here is the conductible cycle consistency loss, which ensures that the warping process is reversible. This constraint prevents the texture from becoming distorted or unrecognizable during the transformation.
3. Try-on Synthesis Module (TSM)
The final stage blends the warped clothing with a coarse image of the person. To ensure the final output remains photo-realistic and recognizable, we introduce a face identity loss (utilizing a pre-trained SphereFaceNet) alongside global and local adversarial losses. This keeps the person's facial features sharp and consistent with the original reference image.
Experimental Evaluation
We evaluated SPG-VTON on two major benchmarks: the MPV (Multi-Pose Virtual Try-on) dataset, which contains over 35,000 images and is known for its real-world "noise" (e.g., handbags and hats), and the DeepFashion dataset.
Quantitative Results
SPG-VTON outperformed the previous state-of-the-art (MG-VTON) across key metrics:
- SSIM (Structural Similarity): Achieved 0.752 on the MPV dataset (vs. 0.744 for MG-VTON).
- IS (Inception Score): Reached 3.243, indicating superior image realism and variety.
Qualitative Results
Visual assessments confirm that SPG-VTON preserves fine details—such as logos, text, and complex patterns—much more effectively than previous methods, particularly in side-view or occluded poses.
Conclusion and Insights
The success of SPG-VTON highlights several key insights for the field of computer vision:
- End-to-End Synergy: Training modules together allows the network to self-correct errors across stages, leading to higher overall quality.
- Semantics as a Prior: Using human parsing maps provides a stronger geometric constraint than RGB pixels alone, forcing the network to respect human anatomy.
- Identity Preservation: Specific losses for facial features are crucial for avoiding the "uncanny valley" effect in virtual try-on tasks.
While SPG-VTON represents a significant advancement, future work will focus on closing the gap between predicted and ground-truth semantic maps and extending the architecture to handle extreme occlusions and natural language-based clothing generation.
Summary updated