TL;DR
This paper introduces TryOn-Refiner, a novel plug-and-play module that significantly enhances the detail reconstruction of virtual try-on results. By leveraging a conditional rectified-flow mechanism and treating coarse try-on images as a starting point instead of random noise, it achieves more accurate and faster refinement.
What Problem Does This Paper Address?
Virtual try-on (VTON) technology is revolutionizing e-commerce by allowing users to visualize themselves in different clothing items digitally. While recent advancements, particularly those based on diffusion models, have improved the realism of VTON, they often struggle with preserving fine-grained details like intricate patterns, text on garments, and subtle facial features. This leads to outputs that, while generally good, lack the sharpness and accuracy of real-world images, hindering their practical application. Existing refinement methods often rely on diffusion models themselves, which can be slow and introduce uncertainty.
Methods
The core innovation is the TryOn-Refiner, a module designed to refine existing VTON outputs. Unlike traditional diffusion-based refinement that starts from random noise, TryOn-Refiner uses the coarse try-on result as a deterministic starting point. This is achieved through a conditional rectified-flow-based mechanism.
Here's a breakdown of the key methodological aspects:
- Conditional Rectified Flow: Instead of a noise-to-image paradigm, TryOn-Refiner employs a flow mapping framework. It learns a direct mapping from the coarse try-on image (with rough details) to the refined image (with accurate details). This deterministic approach avoids the uncertainty inherent in noise-based methods, leading to more stable and precise refinement.
- Plug-and-Play Module: The refiner is designed to be easily integrated with existing VTON methods, acting as a post-processing step.
- Data Construction Pipeline: To generate the necessary paired training data (coarse vs. refined try-on images), the authors propose a pipeline that involves downsampling, a VAE encoding-decoding process to disrupt details, and a super-resolution step.
- Data Smoothing Strategy: To combat "blocking artifacts" (over-sharpened details) that arise from the data construction, they introduce a strategy involving Gaussian low-pass filtering on target images and fine-tuning the VAE decoder.
- Architecture: The framework is built upon the SDXL architecture, utilizing a two-branch design: a main branch for processing the coarse try-on results and a reference branch for the garment information.
Datasets
The paper uses two public benchmarks for training and evaluation:
- VITON-HD: A dataset of 13,679 image pairs of women's upper bodies and garments.
- Dresscode: A larger dataset with 53,792 full-body person and garment pairs across various categories.
The authors also developed a custom data construction pipeline for generating training data specifically for the refinement task, which is crucial for rectified-flow-based methods.
Key Insights and Ideas
- Determinism is Key for Refinement: The core insight is that refinement, unlike initial image generation, doesn't need to start from uncertainty (noise). Treating the coarse try-on result as a fixed starting point allows for a more direct and efficient mapping to the desired refined output.
- Rectified Flow for Efficiency: Rectified flow offers a more direct path between distributions, leading to fewer inference steps (1-10 steps) compared to traditional diffusion-based refinement (30-50 steps).
- Data Quality is Paramount: The proposed data construction pipeline and smoothing strategy highlight the critical role of high-quality, aligned training data for effective refinement. The challenge of artifacts like blocking necessitates careful data preparation.
- Generalizability: The plug-and-play nature of TryOn-Refiner means it can improve the performance of various existing VTON methods, not just a specific architecture.
Key Results
- Improved Detail Reconstruction: TryOn-Refiner consistently improves the quality of try-on results, particularly in preserving intricate details like text, patterns, and textures, as demonstrated in qualitative comparisons.
- Efficiency: The method achieves significant improvements with only 1-10 inference steps, making it much faster than traditional diffusion-based refinement.
- Quantitative Performance: TryOn-Refiner-based methods achieved competitive results, securing strong LPIPS, FID, and KID scores on VITON-HD and Dresscode benchmarks, often surpassing or matching state-of-the-art methods. For instance, when applied to OOTD, it reduced LPIPS from 0.071 to 0.069 and improved SSIM from 0.878 to 0.886.
- Resolution Matters: Experiments showed that higher resolutions (e.g., 1600x1200) yield the best performance, with LPIPS as low as 0.066.
- User Study: A user study indicated that refined results were rated as "good" or "same" in over 95% of cases, with less than 1% worsening.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while improvements are made, some existing SOTA try-on models are already highly effective, making further enhancement challenging.
- Evaluations were conducted at 512x384 resolution, which might obscure finer metric gains.
- The "Same" vote in the user study for simple cases suggests that the impact of the refiner is more pronounced on complex details, which are less frequent in some test sets.
- Future Work:
- The paper suggests that the SDXL framework can be further explored for even better results.
- The authors implicitly suggest that further research into data construction and smoothing strategies could lead to even more robust refinement.
Summary updated