Arbitrary Virtual Try-on Network: Characteristics Representation and Trade-off between Body and Clothing

Yu Liu, Mingbo Zhao, Zhao Zhang, Jicong Fan, Yang Lou, Shuicheng YanICLR 2023 · OpenReview

Overview

The Arbitrary Virtual Try-On Network (AVTON) breaks the "tops-only" limitation prevalent in current virtual try-on systems, allowing users to realistically swap tops, bottoms, or full-body outfits. By introducing a dedicated limb-prediction module and a localized geometric warping technique, AVTON successfully handles "cross-category" swaps—such as changing from long sleeves to a tank top—without leaving the user with missing or distorted limbs.

The Challenge: Beyond Upper-Body Try-On

Most existing virtual try-on (VTO) research focuses exclusively on upper-body clothing. This creates two major bottlenecks:

  1. Category Rigidity: Models trained exclusively on tops cannot handle pants, skirts, or dresses.
  2. The "Missing Limb" Problem: If a user tries on a short-sleeved shirt while wearing a long-sleeved one in their reference photo, the AI often struggles to "fill in" the bare arms. It typically produces blurry artifacts or incorrectly paints clothing textures directly onto the skin.

Furthermore, the standard method for stretching clothes to fit a body—Thin-Plate Spline (TPS)—tends to warp the entire image globally. This often results in unrealistic distortions where a small adjustment in one area causes an unnatural "pull" in another.

The AVTON Architecture

To address these limitations, the authors propose a novel framework consisting of three core components.

The AVTON architecture featuring the Limbs Prediction, Improved Geometric Matching, and Trade-Off Fusion modules.
  • Limbs Prediction Module (LPM): Instead of merely overlaying clothes, this module predicts what the person’s body (arms, legs, skin tone) looks like underneath or outside the target clothing. This is crucial for cross-category tasks (e.g., swapping long pants for skirts) to ensure skin tones and textures remain consistent.
  • Improved Geometric Matching Module (IGMM): The researchers replaced traditional TPS warping with Wendland’s $\Psi$-function. Unlike TPS, which has a "global" influence, the $\Psi$-function is "compactly supported." This allows the model to warp specific parts of a garment (like a sleeve) without distorting the rest of the shirt, preserving the original texture and patterns much more effectively.
  • Trade-Off Fusion Module (TOFM): This GAN-based module uses a dual-UNet structure to balance the characteristics of the warped clothes and the reference person. It specifically addresses artifacts near the neck and joints by "trading off" features between the two inputs, ensuring the final result looks like a cohesive photograph rather than a digital collage.

Datasets and Evaluation

The study utilizes two primary datasets:

  1. VITON-Dataset: The industry standard, containing ~16,000 pairs of frontal-view women and top-wear.
  2. Zalando-Dataset (New): To prove "arbitrary" try-on capabilities, the authors collected a new dataset of ~35,000 images. Unlike VITON, this includes both men and women, and covers three distinct categories: tops, bottoms, and whole outfits (dresses/jumpsuits).

Experimental Results

AVTON demonstrated superior performance across both quantitative metrics and qualitative visual assessments.

Qualitative comparison of AVTON against CP-VTON across top, bottom, and whole-body clothing categories.
  • Quantitative: On the VITON dataset, AVTON achieved an Inception Score (IS) of 3.024, significantly higher than the previous state-of-the-art (ACGPN at 2.829). It also showed marked improvements in structural similarity (SSIM) and perceptual loss (LPIPS).
  • Qualitative: The model excels at "clothing collocation"—putting together an entire outfit sequentially. Visual results show much cleaner edges around the neck and realistic skin generation on arms and legs compared to previous models like CP-VTON.
  • User Study: In a blind test with 40 volunteers, AVTON's results were selected as "most realistic" over 60% of the time when compared to other leading methods.

Limitations and Future Directions

While the paper represents a leap forward in "arbitrary" try-on, some challenges remain:

  • Resolution: The experiments were conducted at a $256 \times 192$ resolution. While standard for research, real-world fashion applications often require high-definition (HD) outputs.
  • Complex Poses: The dataset primarily features frontal views. Handling highly occluded poses (e.g., a person standing sideways or with arms crossed) remains a difficult frontier for the warping modules.
  • Video Integration: The current framework is designed for static images. Extending this logic to video—where temporal consistency of the warped clothes is vital—is a logical next step for future research.

Summary updated

Abstract

Deep learning based virtual try-on system has achieved some encouraging progress recently, but there still remain several big challenges that need to be solved, such as trying on arbitrary clothes of all types, trying on the clothes from one category to another and generating image-realistic results with few artifacts. To handle this issue, we propose the Arbitrary Virtual Try-On Network (AVTON) that is utilized for all-type clothes, which can synthesize realistic try-on images by preserving and trading off characteristics of the target clothes and the reference person. Our approach includes three modules: 1) Limbs Prediction Module, which is utilized for predicting the human body parts by preserving the characteristics of the reference person. This is especially good for handling cross-category try-on task (e.g., long sleeves \(\leftrightarrow\) short sleeves or long pants \(\leftrightarrow\) skirts, etc.), where the exposed arms or legs with the skin colors and details can be reasonably predicted; 2) Improved Geometric Matching Module, which is designed to warp clothes according to the geometry of the target person. We improve the TPS-based warping method with a compactly supported radial function (Wendland's \(\Psi\)-function); 3) Trade-Off Fusion Module, which is to trade off the characteristics of the warped clothes and the reference person. This module is to make the generated try-on images look more natural and realistic based on a fine-tuning symmetry of the network structure. Extensive simulations are conducted and our approach can achieve better performance compared with the state-of-the-art virtual try-on methods.
BibTeX
@inproceedings{liu2023arbitrary,
  author = {Yu Liu and Mingbo Zhao and Zhao Zhang and Jicong Fan and Yang Lou and Shuicheng YAN},
  title = {{Arbitrary Virtual Try-on Network: Characteristics Representation and Trade-off between Body and Clothing}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=d8mr8lKIZ3n},
}
Figure