WAS-VTON: Warping Architecture Search for Virtual Try-on Network

Zhenyu Xie, Xujie Zhang, Fuwei Zhao, Haoye Dong, Michael C. Kampffmeyer, Haonan Yan, Xiaodan Liang

Abstract

Despite recent progress on image-based virtual try-on, current methods are constrained by shared warping networks and thus fail to synthesize natural try-on results when faced with clothing categories that require different warping operations. In this paper, we address this problem by finding clothing category-specific warping networks for the virtual try-on task via Neural Architecture Search (NAS). We introduce a NAS-Warping Module and elaborately design a bilevel hierarchical search space to identify the optimal network-level and operation-level flow estimation architecture. Given the network-level search space, containing different numbers of warping blocks, and the operation-level search space with different convolution operations, we jointly learn a combination of repeatable warping cells and convolution operations specifically for the clothing-person alignment. Moreover, a NAS-Fusion Module is proposed to synthesize more natural final try-on results, which is realized by leveraging particular skip connections to produce better-fused features that are required for seamlessly fusing the warped clothing and the unchanged person part. Extensive experiments demonstrate that our WAS-VTON significantly outperforms the previous fixed-architecture try-on methods with more natural warping results and virtual try-on results.

Introduction

Image-based virtual try-on aims to realistically overlay an in-shop clothing item onto a person's image. The core challenge lies in warping: transforming the flat clothing image to match the person's pose and body shape. Current state-of-the-art methods typically rely on fixed, "one-size-fits-all" architectures for this task. However, a "category-agnostic" design is inherently inefficient. A simple vest requires minimal deformation, whereas long-sleeved shirts or complex skirts involve non-rigid movements. Using the same network for both leads to visual artifacts, blurred textures, or unrealistic silhouettes.

Methodology

The proposed WAS-VTON framework consists of three primary components designed to move beyond fixed architectures:

Partial Parsing Prediction (PPP) Module

This module predicts a semantic map of the target person to guide the initial fit. It utilizes a Geometry Matching Module (GMM) to provide preliminary guidance on how the clothing should align with the body.

NAS-Warping Module

Instead of a static flow-estimation network, this module employs a bilevel hierarchical search space. It searches at the network level (determining the number of warping blocks per cell) and the operation level (selecting optimal convolution types, such as 1x1 or 3x3 depthwise-separable). This allows the system to discover architectures tailored to specific categories, such as "Short Sleeve" vs. "Pants."

NAS-Fusion Module

To seamlessly blend warped clothing with the person's image, the authors search for an optimal U-Net-style architecture. The search focuses on identifying the best skip connections between the encoder and decoder to preserve fine-grained details like skin texture and clothing edges.

The search is executed using a one-shot NAS framework combined with an evolutionary algorithm, training a "supernet" once and efficiently identifying the best sub-networks.

Experimental Results

The model was evaluated on the standard VITON Dataset and an Extended VITON dataset, which includes a wider variety of garments like pants and skirts.

Quantitative Performance

WAS-VTON demonstrated superior performance across key metrics:

  • SSIM (Structural Similarity): Achieved 0.8577 on warped clothes, compared to ACGPN's 0.8099.
  • FID (Fréchet Inception Distance): Reduced the FID to 13.83, nearly halving the scores of previous methods and indicating significantly higher realism.
  • User Preference: In a human evaluation study, 63.07% of participants preferred WAS-VTON over competitors.

Qualitative Analysis

The search results confirmed that different clothing categories benefit from different network complexities. For instance, architectures searched for "Pants" are deeper than those for "Short Sleeves." Furthermore, the NAS-Fusion module discovered that non-standard skip connections across different scales are often more effective at integrating geometric information.

Qualitative results from the proposed WAS-VTON method showing realistic warping and blending.
Comparison results from the ACGPN baseline, showing more artifacts and less precise warping.

Discussion and Conclusion

WAS-VTON is the first framework to successfully apply Neural Architecture Search to virtual try-on. By automatically discovering category-specific warping architectures and optimized fusion layers, it achieves state-of-the-art results in both structural accuracy and visual realism.

While the inference speed is competitive, the initial architecture search is computationally intensive; training the NAS-Warping supernet takes approximately three days. Future work may involve extending this approach to handle extreme poses, multi-garment scenarios, or searching for optimal loss function combinations.

Summary updated

Abstract

Despite recent progress on image-based virtual try-on, current methods are constraint by shared warping networks and thus fail to synthesize natural try-on results when faced with clothing categories that require different warping operations. In this paper, we address this problem by finding clothing category-specific warping networks for the virtual try-on task via Neural Architecture Search (NAS). We introduce a NAS-Warping Module and elaborately design a bilevel hierarchical search space to identify the optimal network-level and operation-level flow estimation architecture. Given the network-level search space, containing different numbers of warping blocks, and the operation-level search space with different convolution operations, we jointly learn a combination of repeatable warping cells and convolution operations specifically for the clothing-person alignment. Moreover, a NAS-Fusion Module is proposed to synthesize more natural final try-on results, which is realized by leveraging particular skip connections to produce better-fused features that are required for seamlessly fusing the warped clothing and the unchanged person part. We adopt an efficient and stable one-shot searching strategy to search the above two modules. Extensive experiments demonstrate that our WAS-VTON significantly outperforms the previous fixed-architecture try-on methods with more natural warping results and virtual try-on results.
BibTeX
@inproceedings{xie2021was,
  author = {Zhenyu Xie and Xujie Zhang and Fuwei Zhao and Haoye Dong and Michael C. Kampffmeyer and Haonan Yan and Xiaodan Liang},
  title = {{WAS-VTON: Warping Architecture Search for Virtual Try-on Network}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475490},
}
Figure