Abstract
Virtual try-on (VTON) has seen significant progress with diffusion models, but current state-of-the-art methods often rely on complex auxiliary modules and heavy preprocessing. CatVTON introduces a streamlined framework that leverages simple spatial concatenation and parameter-efficient training to achieve superior results. By removing redundant components like text encoders and focusing on self-attention layers, CatVTON reduces memory consumption by nearly 50% while improving the preservation of fine-grained garment details and generalization to "in-the-wild" scenarios.
Introduction: The Complexity of Modern VTON
Realistic virtual try-on—the process of overlaying a garment onto a person's image—is a challenging task that requires precise alignment and texture preservation. Recent diffusion-based approaches have become increasingly "bloated," typically requiring:
- Auxiliary Networks: ReferenceNets or heavy image encoders (e.g., CLIP, DINOv2) to inject garment features.
- Extensive Preprocessing: Dependency on external tools for pose estimation, human parsing, and text captioning.
- High Resource Demands: Training hundreds of millions of parameters and requiring significant VRAM for inference.
CatVTON addresses these inefficiencies by questioning the necessity of these extra layers. The authors argue that such complexity often introduces noise and loses fine-grained textures, such as small text or intricate logos.
Methodology: The Power of Simplicity
The core philosophy of CatVTON is that "concatenation is all you need." Instead of complex feature-injection schemes, the model utilizes spatial concatenation.
Architecture and Simplified UNet
CatVTON concatenates the garment image and the person image side-by-side, feeding them into a single, standard denoising UNet. This allows the model to learn relationships between the garment and the person within a shared latent space. Furthermore, the authors removed the text encoder and all cross-attention modules, as text descriptions are redundant when the actual garment image is provided. This significantly lightens the network.
Parameter-Efficient Training
The research identifies self-attention layers as the critical component for the try-on task. By training only these layers (approximately 50M parameters) and freezing the rest of the UNet, the model adapts to the VTON task while preserving the powerful generative priors of the original diffusion model.
Mask-Free Pipeline
To eliminate the need for human parsing during inference, CatVTON employs a two-stage training process. A mask-based model is first used to generate high-quality synthetic data, which then trains a "mask-free" version capable of automatically identifying person and garment regions.
Experimental Results
CatVTON was evaluated on major benchmarks, including VITON-HD, DressCode, and DeepFashion. Despite being trained on a relatively small dataset of ~73K samples, it demonstrates remarkable generalization.
Efficiency and Performance
- Reduced Parameters: Uses only 899M total parameters and 49.57M trainable parameters, compared to 2B+ in models like IDM-VTON.
- Memory and Speed: Inference is 2-5x faster, with a 49% reduction in memory usage, enabling execution on consumer-grade GPUs with less than 8GB VRAM.
- Quantitative Superiority: Achieved the lowest Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) on the VITON-HD dataset.
Visual Fidelity
CatVTON excels at preserving complex textures and handling challenging "in-the-wild" scenarios involving messy backgrounds or unusual poses.
Key Insights
- Spatial vs. Channel Concatenation: Spatial concatenation allows self-attention to find correspondences across the garment and person more effectively than stacking channels.
- Redundancy of Text: Removing text-conditioning modules improved performance and reduced parameters, as text prompts can act as noise in image-based VTON.
- Synthetic Data for Autonomy: Using model-generated data allows the system to function without external segmentation masks, a significant step toward real-world deployment.
Limitations and Future Directions
While efficient, CatVTON faces some limitations. The standard VAE bottleneck can occasionally lose tiny details or cause slight color shifts. Additionally, while the mask-free version is robust, the highest quality results still benefit from accurate masks. Future work may explore more advanced backbones, such as Diffusion Transformers (DiT), to further enhance high-resolution performance.
Summary updated