UVIRT - Unsupervised Virtual Try-on Using Disentangled Clothing and Person Features

Hideki Tsunashima, Kosuke Arase, Antony Lam, Hirokatsu Kataoka

TL;DR

This paper introduces UVIRT, a novel unsupervised virtual try-on method that disentangles clothing and person features to realistically superimpose garments onto a target person without requiring any annotated data or pre-trained models. This breakthrough significantly reduces the cost and complexity of virtual try-on, making it applicable to real-world e-commerce scenarios, particularly challenging C2C marketplaces.

What Problem Does This Paper Address?

The booming online apparel market, especially C2C platforms, faces a significant hurdle: customers cannot try on clothes before buying. Existing virtual try-on solutions are predominantly supervised, relying heavily on costly annotated data like semantic segmentation masks and paired images of people wearing specific outfits. This annotation burden makes them impractical for large-scale, dynamic C2C environments where items are constantly added and variations are immense. Even "weakly-supervised" methods still require auxiliary information from pre-trained models, adding latency and complexity. The core problem is the need for a virtual try-on system that is effective, cost-efficient, and adaptable to real-world, unannotated data.

Methods

UVIRT tackles this problem by employing an unsupervised approach based on disentangled representation learning. The core idea is to separate the "content" (clothing) and "style" (person and background) features from input images.

  1. Feature Disentanglement: UVIRT uses two encoders to extract content features (clothing) and style features (person/background) from both a person image and a clothing image.
  2. Feature Exchange: Crucially, the method then exchanges these disentangled content features between the person and clothing domains. For instance, the clothing features from the clothing image are combined with the person's style features.
  3. Reconstruction: Finally, decoders reconstruct the new image, effectively warping the clothing onto the person. This process is achieved using a technique called Group-wise Deep Whitening-and-Coloring Transformation (GDWCT), which helps to disentangle the features more effectively than standard methods like Adaptive Instance Normalization (AdaIN).

The entire process is unsupervised, meaning it doesn't need any ground truth labels, paired images, or pre-trained models for tasks like pose estimation or segmentation.

Datasets

  • MPV Dataset: A standard dataset for virtual try-on, containing paired images of people and clothing. This was used for evaluating UVIRT against supervised methods in a controlled setting.
  • C2C Market Dataset: A novel dataset collected by the authors from a real-world C2C marketplace (Mercari). This dataset consists of unpaired, in-the-wild images of clothing, which are significantly more challenging due to variations in camera types, lighting, and image alignment (e.g., clothes are often shown hanging or folded). This dataset is key to demonstrating UVIRT's real-world applicability.

Key Insights and Ideas

  • Unsupervised Disentanglement is Key: The paper demonstrates that by effectively disentangling clothing and person features in a latent space, it's possible to perform virtual try-on without any supervision. This is a significant departure from prior work.
  • C2C Marketplaces are a Unique Challenge: The authors highlight that real-world C2C data is inherently unaligned and diverse, making it unsuitable for traditional supervised methods. UVIRT's ability to handle this is a major advantage.
  • GDWCT for Better Disentanglement: The use of GDWCT is presented as a crucial component for achieving better separation of content and style features, leading to more realistic try-on results.
  • Speed Advantage: By avoiding the need for auxiliary networks like pose estimators or parsers, UVIRT achieves a substantial speedup in testing time compared to supervised methods.

Key Results

  • C2C Performance: UVIRT significantly outperforms a supervised method (CP-VTON) on the challenging C2C market dataset, demonstrating its effectiveness in real-world, unaligned scenarios.
  • MPV Performance: On the MPV dataset (with paired images), UVIRT achieves comparable results to conventional supervised methods, showcasing its robustness.
  • Speed: UVIRT is approximately 13 times faster than CP-VTON during testing, making it more practical for real-time applications.
  • Qualitative Results: Visual comparisons show UVIRT can warp clothing onto people effectively, even with unaligned images, though it sometimes alters the person's pose.

Limitations and Future Work

  • Pose and Face Reconstruction: The authors acknowledge that UVIRT can alter the person's pose and that face reconstruction is not always perfect. They also note that the generated images can sometimes show distortions or color artifacts, particularly when the person and clothing features are not perfectly disentangled.
  • Feature Entanglement: A key limitation identified is that the disentangled person and clothing features might not be semantically orthogonal, leading to entanglement.
  • Future Work: The authors plan to address these limitations by exploring unsupervised disentanglement techniques from generative models to better separate features and improve reconstruction quality. They also aim to resolve issues with person distortion and face reconstruction.

Summary updated

Abstract

Virtual Try-on is the ability to realistically superimpose clothing onto a target person. Due to its importance to the multi-billion dollar e-commerce industry, the problem has received significant attention in recent years. To date, most virtual try-on methods have been supervised approaches, namely using annotated data, such as clothes parsing semantic segmentation masks and paired images. These approaches incur a very high cost in annotation. Even existing weakly-supervised virtual try-on methods still use annotated data or pre-trained networks as auxiliary information and the costs of the annotation are still significantly high. Plus, the strategy using pre-trained networks is not appropriate in the practical scenarios due to latency. In this paper we propose Unsupervised VIRtual Try-on using disentangled representation (UVIRT). After UVIRT extracts a clothes and a person feature from a person image and a clothes image respectively, it exchanges a clothes and a person feature. Finally, UVIRT achieve virtual try-on. This is all achieved in an unsupervised manner so UVIRT has the advantage that it does not require any annotated data, pre-trained networks nor even category labels. In the experiments, we qualitatively and quantitatively compare between supervised methods and our UVIRT method on the MPV dataset (which has paired images) and on a Consumer-to-Consumer (C2C) marketplace dataset (which has unpaired images). As a result, UVIRT outperform the supervised method on the C2C marketplace dataset, and achieve comparable results on the MPV dataset, which has paired images in comparison with the conventional supervised method.
BibTeX
@inproceedings{tsunashima2020uvirt,
  author = {Hideki Tsunashima and Kosuke Arase and Antony Lam and Hirokatsu Kataoka},
  title = {{UVIRT - Unsupervised Virtual Try-on Using Disentangled Clothing and Person Features}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=CEYEY6PWJhc},
}
Figure