GC-VTON: Predicting Globally Consistent and Occlusion Aware Local Flows With Neighborhood Integrity Preservation for Virtual Try-On

Hamza Rawal, Muhammad Junaid Ahmad, Farooq ZamanWACV 2024

TL;DR

This paper introduces GC-VTON, a novel virtual try-on method that disentangles global garment alignment from local texture preservation to achieve more realistic and artifact-free results. It tackles challenges like occlusions and texture distortion by introducing a visibility mask and a new regularization loss (NIPR) that preserves local neighborhood integrity.

What Problem Does This Paper Address?

Virtual try-on aims to realistically render a person wearing a new garment. A critical step in this process is "garment warping," where the input garment is deformed to match the target person's pose and body shape. Existing methods often struggle with this task, leading to several artifacts:

  • Global vs. Local Conflicts: Optimizing a single network for both global boundary alignment (making the garment fit the person's overall shape) and local texture preservation (keeping the garment's fabric details sharp and undistorted) is a difficult trade-off. This often results in either misaligned boundaries or distorted textures.
  • Occlusions: Body parts (like arms or hands) or other clothing items can obscure parts of the garment. Standard flow prediction struggles to handle these occlusions gracefully, often leading to tearing or unnatural squeezing of the fabric around the occluded areas.
  • Texture Distortion: Even without occlusions, dense flow fields can lead to texture squeezing or stretching, especially in challenging scenarios like tucked-in garments, making the try-on look unnatural. Existing regularization losses, which aim for smooth flows, are often too local and cannot effectively identify or penalize these specific texture integrity violations.

Methods

GC-VTON proposes a multi-module approach to address these challenges:

  • Disentangled Warping Modules (GlobalNet & LocalNet): The core innovation is separating the warping task into two specialized modules.
  • LocalNet: Focuses on preserving local texture details by taking the original garment image as input. It predicts local flow fields and a body-part visibility mask.
  • GlobalNet: Handles global boundary alignment, aiming to match the garment's outline to the person's pose. It's designed to be texture-agnostic, taking only garment masks and person representations as input. This separation prevents a "tug-of-war" between the two objectives.
  • Consistency Loss: A loss function is introduced to ensure that the outputs of LocalNet and GlobalNet are harmonized, leading to locally consistent warps that also align globally.
  • Occlusion Handling (Visibility Mask): LocalNet predicts a body-part visibility mask. This mask is used to explicitly mask out occluded regions in the warped garment, preventing the flow prediction network from attempting to warp areas that are not visible. This mask is also fed to the generator for better synthesis.
  • Neighborhood Integrity Preserving Regularization (NIPR): A novel regularization loss is proposed to specifically penalize texture integrity violations (squeezing and stretching) in local neighborhoods. Unlike previous methods that focus solely on local pixel distances, NIPR considers global garment changes and the type of artifact, applying more appropriate penalties.

Datasets

The paper uses the VITON-HD dataset for training and evaluation. This dataset contains frontal-view images of women wearing upper garments, with corresponding garment images.

Key Insights and Ideas

  • Decomposition is Key: The most significant insight is that disentangling the global alignment and local texture preservation tasks into separate modules is crucial for achieving high-quality warps. Forcing a single network to do both leads to compromises.
  • Visibility Masks for Occlusions: Explicitly predicting and using a visibility mask is a simple yet effective way to handle occlusions, preventing the warping network from generating nonsensical flows in occluded areas.
  • Artifact-Specific Regularization: Standard smoothness losses are insufficient because they don't understand why a warp is bad (squeezing vs. stretching) or how local distortions relate to global garment changes. NIPR's targeted approach is a more effective way to preserve texture integrity.
  • Global Context for Local Warping: NIPR's success highlights the importance of incorporating global garment context when regularizing local flow fields.

Key Results

  • Quantitative Performance: GC-VTON outperforms existing state-of-the-art methods (VITON-HD, HR-VTON, FS-VTON) across multiple metrics, including SSIM, FID, and LPIPS, on the VITON-HD dataset.
  • Human Evaluation: GC-VTON was preferred by human evaluators 54.3% of the time in side-by-side comparisons, indicating its superior realism.
  • Qualitative Improvements: Visual results demonstrate GC-VTON's ability to produce natural-looking images with significantly fewer artifacts, particularly in challenging areas like texture sharpness, linearity, and handling of tucked-in styles and occlusions. Ablation studies show that each component (GlobalNet, Occlusion Handling, NIPR) contributes to these improvements.

Limitations and Future Work

  • Limitations: The paper acknowledges that while NIPR is effective, it's still a regularization loss applied to flow. The fundamental limitations of 2D image-based try-on, such as the inability to fully capture 3D garment dynamics or complex body shapes, remain. The method is primarily tested on upper garments.
  • Future Work: The authors suggest that their approach can be extended to handle full-body try-on. Further research could explore more sophisticated ways to integrate 3D information or more advanced generative models to further enhance realism and handle more complex scenarios.

Summary updated

BibTeX
@inproceedings{rawal2024gc,
  author = {Hamza Rawal and Muhammad Junaid Ahmad and Farooq Zaman},
  title = {{GC-VTON: Predicting Globally Consistent and Occlusion Aware Local Flows With Neighborhood Integrity Preservation for Virtual Try-On}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure