An Efficient Style Virtual Try on Network

Shanchen Pang, Xixi Tao, Yukun DongarXiv 2021 · arXiv

TL;DR

This paper introduces Style-VTON, a novel three-stage network for virtual clothing try-on that excels at preserving garment texture and pattern while allowing for stylistic customization. It addresses the limitations of existing methods by employing UV mapping for efficient texture transfer and a style generation network for personalized adjustments, leading to more realistic and user-satisfying virtual try-on experiences.

What Problem Does This Paper Address?

Traditional virtual try-on methods often struggle to accurately render clothing textures and patterns, especially when dealing with complex human poses and occlusions. Existing approaches, while improving, can lead to distorted garments, loss of detail, and a lack of personalization, hindering their adoption in real-world e-commerce applications. The core challenge is to seamlessly transfer a target garment onto a reference person, maintaining the garment's original appearance and allowing users to stylize it to their preference, all while handling the inherent complexities of human shape and posture.

Methods

Style-VTON employs a three-stage approach:

  1. Condition Analysis Network: This stage takes a user image, target clothing image, and a posture heat map as input. It generates a detailed human parsing map, which is crucial for understanding body shape and key points. This network leverages DenseNet for feature extraction and a conditional adversarial network (CGAN) framework.
  2. Pix22DSurf Neural Mapping: This is the core innovation for texture transfer. It utilizes UV mapping to establish a one-to-one correspondence between garment pixels and user body pixels. This allows for efficient, real-time mapping of patterns and textures onto deformed silhouettes, preserving details and ensuring spatial rationality. This stage uses a custom non-rigid pixel-to-pixel registration method.
  3. Style Generation Network: This final stage allows for user-driven stylistic adjustments. It enables users to modify texture, color, and style elements of the generated try-on image, enhancing user experience and offering diverse clothing collocation options. This is achieved through an editing module that manipulates texture and shape features separately.

Datasets

  • VITON dataset: Expanded and supplemented with data from Zalando, Tom Tailor, and Jack & Jones, totaling approximately 17,000 image pairs (14,875 for training, 2,125 for testing). This dataset focuses on virtual try-on for tops.
  • ChicTopia10K dataset: Used for training a fashion classifier for minimal editing. 15,930 images were used for generator training, and 12,744 for classifier training, with 3,240 unfashionable examples for evaluation.

Key Insights and Ideas

  • UV Mapping for Texture Preservation: The paper highlights the effectiveness of UV mapping in establishing dense correspondences between garment and body surfaces, which is critical for accurately transferring complex textures and patterns without distortion.
  • Three-Stage Design for Comprehensive Try-On: The modular, three-stage approach (analysis, mapping, stylization) allows each component to focus on a specific aspect of the virtual try-on problem, leading to a more robust and versatile solution.
  • Personalization through Style Generation: The inclusion of a style generation network is a significant step towards making virtual try-on truly interactive and personalized, moving beyond simple garment transfer to user-driven fashion choices.
  • Addressing Occlusion and Deformation: The fine-grained semantic segmentation and UV mapping techniques are specifically designed to mitigate issues caused by body occlusions and large-scale deformations, which are common challenges in this field.

Key Results

  • Quantitative Evaluation: Style-VTON achieved strong results in terms of IS (Image Similarity) and SSIM (Structural Similarity Index Measure), with scores of 2.887 and 0.859 respectively, outperforming VITON, VTNFP, and CP-VTON.
  • Qualitative Evaluation: User studies (A/B tests) showed that Style-VTON's generated images were preferred by volunteers significantly more often than those from VITON (77.87%) and CP-VTON (81.38%), indicating superior visual quality and authenticity.
  • Efficiency: The Pix22DSurf method is noted for significantly increasing processing speed for texture mapping.
  • Stylization: The stylized results were perceived as having a slight fashion improvement or little change, suggesting successful and subtle stylistic adjustments.

Limitations and Future Work

  • Dataset Limitations: The paper acknowledges the difficulty in obtaining full-body images and the low resolution of some collected images, leading to a focus on virtual try-on for tops.
  • Future Work: The authors plan to expand the training data source to include a wider range of social media platforms. They also aim to generate styles tailored to personal preferences or special occasions, further enhancing the personalization aspect and reducing redundancy in the garment industry.

Summary updated

BibTeX
@misc{pang2021efficient,
  author = {Shanchen Pang and Xixi Tao and Yukun Dong},
  title = {{An Efficient Style Virtual Try on Network}},
  year = {2021},
  eprint = {2105.13183},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.13183},
}
Figure