TL;DR
This paper introduces C-VTON, a novel network for image-based virtual try-on that significantly improves the realism and accuracy of transferring clothing items to subjects in images. It achieves this by employing a context-aware geometric matching module and a powerful image generator that leverages various contextual cues, leading to state-of-the-art results even in challenging scenarios like difficult poses and self-occlusions.
What Problem Does This Paper Address?
The fashion industry is increasingly leveraging e-commerce, and virtual try-on (VTO) technologies are crucial for enhancing the online shopping experience. Image-based VTO, which doesn't require specialized hardware, aims to realistically replace a piece of clothing in an input image with a target garment. However, existing methods often struggle with diverse input image characteristics, leading to artifacts, inaccurate garment alignment, and poor preservation of on-shirt details, especially in challenging poses or with self-occlusions. This limits the believability and utility of current virtual fitting room solutions.
Methods
C-VTON employs a two-stage pipeline:
- Body-Part Geometric Matcher (BPGM): This novel module is designed to align the target clothing with the subject's pose. Unlike previous methods that relied on complex person representations, BPGM uses only body-part segmentations (derived from DensePose) as input. This simplifies the process and makes it more robust to variations in pose and occlusions. The BPGM estimates a Thin-Plate Spline (TPS) transformation to warp the target clothing. Its training involves three loss functions:
- Shape Loss ($L_{shp}$): Ensures the warped clothing matches the subject's body shape.
- Appearance Loss ($L_{app}$): Encourages visual similarity between the warped clothing and the input image within the body area.
- Perceptual Loss ($L_{vgg}$): Preserves semantic content and on-shirt graphics.
- Context-Aware Generator (CAG): This component synthesizes the final virtual try-on image. It takes the warped clothing from BPGM, along with other contextual information (body segmentations, masked input image, target clothing), as input. The CAG is a residual network augmented with Context-Aware Normalization (CAN) layers. These CAN layers adaptively normalize activations based on the input context, allowing for spatially normalized synthesis at different resolutions and ensuring critical information is propagated effectively. The CAG is trained using:
- Perceptual Loss ($L_{per}$): To match the semantic content of the generated image to the reference input.
- Three Adversarial Losses: * Segmentation Discriminator ($D_{seg}$): Ensures realistic body-part generation.
- Matching Discriminator ($D_{mth}$): Verifies that the target garment correctly corresponds to the clothing being worn.
- Patch Discriminator ($D_{ptc}$): Focuses on the realistic appearance of local patches on key body parts.
Datasets
The paper uses two standard datasets for training and evaluation:
- VITON: A popular dataset for virtual try-on, consisting of subject images and corresponding target clothing.
- MPV: Another dataset with a higher degree of appearance variability, including differences in zoom and viewpoint.
The authors filtered out duplicate images from both datasets for fair comparison.
Key Insights and Ideas
- Body-part segmentation is sufficient for robust geometric matching: The paper demonstrates that relying solely on body-part segmentations for pose alignment is effective and more robust than using complex, clothing-agnostic person representations. This simplifies the pipeline and improves performance in challenging poses.
- Context-Aware Normalization (CAN) is key for synthesis: The CAN layers in the generator are crucial for effectively leveraging diverse contextual information (pose, segmentation, etc.) to produce realistic and detailed try-on results.
- Multi-faceted adversarial training: Using three specialized discriminators (segmentation, matching, and patch) allows for a more comprehensive evaluation and improvement of different aspects of the generated image, leading to higher realism.
- Ablation studies highlight component importance: The study shows that while all components contribute, the CAN operations have the most significant impact on overall performance, followed by the BPGM. The discriminators, though contributing less to quantitative scores, are vital for visual quality and artifact reduction.
Key Results
- State-of-the-art performance: C-VTON significantly outperforms competing models on both VITON and MPV datasets in terms of FID and LPIPS scores. For instance, on VITON, it reduces FID by 28.2% and LPIPS by 53.6% compared to the runner-up.
- Improved visual quality: Qualitative comparisons show C-VTON generates more convincing results, particularly in synthesizing arms and hands, preserving on-shirt graphics, and maintaining garment shape, even in difficult poses.
- Human perceptual study: C-VTON was overwhelmingly preferred by human raters in a perceptual study, indicating its superior realism and quality compared to other methods.
- Ablation study findings: Removing CAN operations led to the largest degradation in performance, while removing discriminators resulted in more noticeable visual artifacts despite smaller quantitative score changes.
Limitations and Future Work
The authors acknowledge several limitations:
- Masking procedure issues: Imperfections in the masking procedure for generating the image context can lead to artifacts.
- Loose clothing: The model struggles with very loose clothing in the input images.
- Front/Back differentiation: C-VTON cannot differentiate between the front and back of a target garment.
These limitations can result in less convincing results, such as blurry clothing edges, soft garment edges, and improperly rendered neck areas. The paper suggests that these limitations are also present in competing models. Future work could focus on addressing these specific challenges to further enhance the realism and applicability of virtual try-on systems.
Summary updated