TL;DR
This paper introduces UVIRT, a novel unsupervised virtual try-on method that disentangles clothing and person features to realistically superimpose garments onto a target person without requiring any annotated data or pre-trained models. This breakthrough significantly reduces the cost and complexity of virtual try-on, making it applicable to real-world e-commerce scenarios, particularly challenging C2C marketplaces.
What Problem Does This Paper Address?
The booming online apparel market, especially C2C platforms, faces a significant hurdle: customers cannot try on clothes before buying. Existing virtual try-on solutions are predominantly supervised, relying heavily on costly annotated data like semantic segmentation masks and paired images of people wearing specific outfits. This annotation burden makes them impractical for large-scale, dynamic C2C environments where items are constantly added and variations are immense. Even "weakly-supervised" methods still require auxiliary information from pre-trained models, adding latency and complexity. The core problem is the need for a virtual try-on system that is effective, cost-efficient, and adaptable to real-world, unannotated data.
Methods
UVIRT tackles this problem by employing an unsupervised approach based on disentangled representation learning. The core idea is to separate the "content" (clothing) and "style" (person and background) features from input images.
- Feature Disentanglement: UVIRT uses two encoders to extract content features (clothing) and style features (person/background) from both a person image and a clothing image.
- Feature Exchange: Crucially, the method then exchanges these disentangled content features between the person and clothing domains. For instance, the clothing features from the clothing image are combined with the person's style features.
- Reconstruction: Finally, decoders reconstruct the new image, effectively warping the clothing onto the person. This process is achieved using a technique called Group-wise Deep Whitening-and-Coloring Transformation (GDWCT), which helps to disentangle the features more effectively than standard methods like Adaptive Instance Normalization (AdaIN).
The entire process is unsupervised, meaning it doesn't need any ground truth labels, paired images, or pre-trained models for tasks like pose estimation or segmentation.
Datasets
- MPV Dataset: A standard dataset for virtual try-on, containing paired images of people and clothing. This was used for evaluating UVIRT against supervised methods in a controlled setting.
- C2C Market Dataset: A novel dataset collected by the authors from a real-world C2C marketplace (Mercari). This dataset consists of unpaired, in-the-wild images of clothing, which are significantly more challenging due to variations in camera types, lighting, and image alignment (e.g., clothes are often shown hanging or folded). This dataset is key to demonstrating UVIRT's real-world applicability.
Key Insights and Ideas
- Unsupervised Disentanglement is Key: The paper demonstrates that by effectively disentangling clothing and person features in a latent space, it's possible to perform virtual try-on without any supervision. This is a significant departure from prior work.
- C2C Marketplaces are a Unique Challenge: The authors highlight that real-world C2C data is inherently unaligned and diverse, making it unsuitable for traditional supervised methods. UVIRT's ability to handle this is a major advantage.
- GDWCT for Better Disentanglement: The use of GDWCT is presented as a crucial component for achieving better separation of content and style features, leading to more realistic try-on results.
- Speed Advantage: By avoiding the need for auxiliary networks like pose estimators or parsers, UVIRT achieves a substantial speedup in testing time compared to supervised methods.
Key Results
- C2C Performance: UVIRT significantly outperforms a supervised method (CP-VTON) on the challenging C2C market dataset, demonstrating its effectiveness in real-world, unaligned scenarios.
- MPV Performance: On the MPV dataset (with paired images), UVIRT achieves comparable results to conventional supervised methods, showcasing its robustness.
- Speed: UVIRT is approximately 13 times faster than CP-VTON during testing, making it more practical for real-time applications.
- Qualitative Results: Visual comparisons show UVIRT can warp clothing onto people effectively, even with unaligned images, though it sometimes alters the person's pose.
Limitations and Future Work
- Pose and Face Reconstruction: The authors acknowledge that UVIRT can alter the person's pose and that face reconstruction is not always perfect. They also note that the generated images can sometimes show distortions or color artifacts, particularly when the person and clothing features are not perfectly disentangled.
- Feature Entanglement: A key limitation identified is that the disentangled person and clothing features might not be semantically orthogonal, leading to entanglement.
- Future Work: The authors plan to address these limitations by exploring unsupervised disentanglement techniques from generative models to better separate features and improve reconstruction quality. They also aim to resolve issues with person distortion and face reconstruction.
Summary updated