FW-GAN: Flow-Navigated Warping GAN for Video Virtual Try-On

Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bowen Wu, Bing-Cheng Chen, Jian Yin

FW-GAN: Flow-Navigated Warping GAN for Video Virtual Try-On

Introduction

Video virtual try-on represents a significant leap from static image-based systems, aiming to provide a realistic preview of how garments move and drape on a person in motion. While image-based methods have matured, extending them to video introduces severe challenges in spatiotemporal consistency. Naive frame-by-frame application often results in "texture flickering," where clothing patterns jump inconsistently between frames, failing to adhere to the body's natural movement.

The FW-GAN Framework

To address these issues, the Flow-navigated Warping GAN (FW-GAN) introduces a multi-stage architecture designed for temporal coherence and high-fidelity texture preservation. The core of the system is the Warping Net. Rather than generating clothing from scratch—which often blurs intricate details—the Warping Net learns a transformation grid to deform a flat image of the target garment to match the person's specific pose.

Target garment used as input for the Warping Net.

To maintain structural integrity, the model employs a Parsing Constraint Loss. This utilizes a human parser to ensure the synthesized body parts and clothing align with the expected anatomical layout for a given pose. This structural guidance is essential for handling complex movements and occlusions.

Pose estimation at t=3, providing the structural guidance for the Parsing Constraint Loss.

Furthermore, a Flow-Guided Fusion Module leverages optical flow to warp previous synthesized frames into the current one. This "temporal memory" ensures that pixels move logically across the sequence, significantly reducing flickering. The training is supervised by a dual-discriminator setup: a Frame Discriminator for per-frame realism and a Flow-embedding Discriminator to enforce physically plausible motion.

The VVT Dataset and Evaluation

The authors introduced the VVT (Video Virtual Try-on) Dataset to benchmark this task. Comprising 791 high-quality videos (approx. 190,000 frames) of fashion models, it offers a diverse range of viewpoints and complex movements compared to previous static datasets. Each video is paired with a specific person image and a target clothes image, enabling supervised training.

Quantitatively, FW-GAN achieves a Frechet Inception Distance (FID) of 7.05, a substantial improvement over the baseline CP-VTON's score of 32.35. Qualitative assessments and human studies further validate the approach, with users preferring FW-GAN's results in over 88% of comparisons due to its ability to maintain sharp logos and consistent textures during movement.

Conclusion and Future Directions

FW-GAN demonstrates that combining explicit warping with flow-guided temporal fusion can overcome the primary hurdles of video virtual try-on. While the system excels on clean backgrounds and standard tops, future work will focus on "in-the-wild" scenarios with complex backgrounds and a wider variety of garment types, such as layered outfits and full-body silhouettes.

Summary updated

Abstract

Beyond current image-based virtual try-on systems that have attracted increasing attention, we move a step forward to developing a video virtual try-on system that precisely transfers clothes onto the person and generates visually realistic videos conditioned on arbitrary poses. Besides the challenges in image-based virtual try-on (e.g., clothes fidelity, image synthesis), video virtual try-on further requires spatiotemporal consistency. Directly adopting existing image-based approaches often fails to generate coherent video with natural and realistic textures. In this work, we propose Flow-navigated Warping Generative Adversarial Network (FW-GAN), a novel framework that learns to synthesize the video of virtual try-on based on a person image, the desired clothes image, and a series of target poses. FW-GAN aims to synthesize the coherent and natural video while manipulating the pose and clothes. It consists of: (i) a flow-guided fusion module that warps the past frames to assist synthesis, which is also adopted in the discriminator to help enhance the coherence and quality of the synthesized video; (ii) a warping net that is designed to warp clothes image for the refinement of clothes textures; (iii) a parsing constraint loss that alleviates the problem caused by the misalignment of segmentation maps from images with different poses and various clothes. Experiments on our newly collected dataset show that FW-GAN can synthesize high-quality video of virtual try-on and significantly outperforms other methods both qualitatively and quantitatively.
BibTeX
@inproceedings{dong2019fw,
  author = {Haoye Dong and Xiaodan Liang and Xiaohui Shen and Bowen Wu and Bing-Cheng Chen and Jian Yin},
  title = {{FW-GAN: Flow-Navigated Warping GAN for Video Virtual Try-On}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00125},
}
Figure