Overview
The Arbitrary Virtual Try-On Network (AVTON) breaks the "tops-only" limitation prevalent in current virtual try-on systems, allowing users to realistically swap tops, bottoms, or full-body outfits. By introducing a dedicated limb-prediction module and a localized geometric warping technique, AVTON successfully handles "cross-category" swaps—such as changing from long sleeves to a tank top—without leaving the user with missing or distorted limbs.
The Challenge: Beyond Upper-Body Try-On
Most existing virtual try-on (VTO) research focuses exclusively on upper-body clothing. This creates two major bottlenecks:
- Category Rigidity: Models trained exclusively on tops cannot handle pants, skirts, or dresses.
- The "Missing Limb" Problem: If a user tries on a short-sleeved shirt while wearing a long-sleeved one in their reference photo, the AI often struggles to "fill in" the bare arms. It typically produces blurry artifacts or incorrectly paints clothing textures directly onto the skin.
Furthermore, the standard method for stretching clothes to fit a body—Thin-Plate Spline (TPS)—tends to warp the entire image globally. This often results in unrealistic distortions where a small adjustment in one area causes an unnatural "pull" in another.
The AVTON Architecture
To address these limitations, the authors propose a novel framework consisting of three core components.
- Limbs Prediction Module (LPM): Instead of merely overlaying clothes, this module predicts what the person’s body (arms, legs, skin tone) looks like underneath or outside the target clothing. This is crucial for cross-category tasks (e.g., swapping long pants for skirts) to ensure skin tones and textures remain consistent.
- Improved Geometric Matching Module (IGMM): The researchers replaced traditional TPS warping with Wendland’s $\Psi$-function. Unlike TPS, which has a "global" influence, the $\Psi$-function is "compactly supported." This allows the model to warp specific parts of a garment (like a sleeve) without distorting the rest of the shirt, preserving the original texture and patterns much more effectively.
- Trade-Off Fusion Module (TOFM): This GAN-based module uses a dual-UNet structure to balance the characteristics of the warped clothes and the reference person. It specifically addresses artifacts near the neck and joints by "trading off" features between the two inputs, ensuring the final result looks like a cohesive photograph rather than a digital collage.
Datasets and Evaluation
The study utilizes two primary datasets:
- VITON-Dataset: The industry standard, containing ~16,000 pairs of frontal-view women and top-wear.
- Zalando-Dataset (New): To prove "arbitrary" try-on capabilities, the authors collected a new dataset of ~35,000 images. Unlike VITON, this includes both men and women, and covers three distinct categories: tops, bottoms, and whole outfits (dresses/jumpsuits).
Experimental Results
AVTON demonstrated superior performance across both quantitative metrics and qualitative visual assessments.
- Quantitative: On the VITON dataset, AVTON achieved an Inception Score (IS) of 3.024, significantly higher than the previous state-of-the-art (ACGPN at 2.829). It also showed marked improvements in structural similarity (SSIM) and perceptual loss (LPIPS).
- Qualitative: The model excels at "clothing collocation"—putting together an entire outfit sequentially. Visual results show much cleaner edges around the neck and realistic skin generation on arms and legs compared to previous models like CP-VTON.
- User Study: In a blind test with 40 volunteers, AVTON's results were selected as "most realistic" over 60% of the time when compared to other leading methods.
Limitations and Future Directions
While the paper represents a leap forward in "arbitrary" try-on, some challenges remain:
- Resolution: The experiments were conducted at a $256 \times 192$ resolution. While standard for research, real-world fashion applications often require high-definition (HD) outputs.
- Complex Poses: The dataset primarily features frontal views. Handling highly occluded poses (e.g., a person standing sideways or with arms crossed) remains a difficult frontier for the warping modules.
- Video Integration: The current framework is designed for static images. Extending this logic to video—where temporal consistency of the warped clothes is vital—is a logical next step for future research.
Summary updated