Abstract
Given two images depicting a person and a garment worn by another person, our goal is to generate a visualization of how the garment might look on the input person. A key challenge is to synthesize a photorealistic detail-preserving visualization of the garment, while warping the garment to accommodate a significant body pose and shape change across the subjects. Previous methods either focus on garment detail preservation without effective pose and shape variation, or allow tryon with the desired shape and pose but lack garment details. In this paper, we propose a diffusion-based architecture that unifies two UNets (referred to as Parallel-UNet), which allows us to preserve garment details and warp the garment for significant pose and body change in a single network. The key ideas behind Parallel-UNet include: 1) garment is warped implicitly via a cross attention mechanism, 2) garment warp and person blend happen as part of a unified process as opposed to a sequence of two separate tasks. Experimental results indicate that TryOnDiffusion achieves state-of-the-art performance both qualitatively and quantitatively.
Introduction
Virtual apparel try-on—visualizing how a specific garment looks on a specific person—is a long-standing challenge in computer vision. The core difficulty lies in "non-rigid warping": the garment must be deformed to fit a target body shape and pose while maintaining its original patterns, textures, and labels without looking distorted.
Previous methods typically followed a two-step "warp-then-blend" pipeline. They first estimated pixel displacements (like optical flow) to move the clothes and then used a second network to blend them onto the person. This often failed during heavy occlusions or extreme pose changes, leading to "ghosting" artifacts or blurred textures. Other generative approaches (like GANs) produced high-quality images but often lost fine-grained garment details because they compressed the information into a low-dimensional latent space. TryOnDiffusion introduces a novel "Parallel-UNet" architecture that achieves state-of-the-art virtual try-on results at 1024×1024 resolution.
Methodology: The Parallel-UNet Architecture
The authors propose TryOnDiffusion, which moves away from explicit flow estimation in favor of a unified diffusion process. The centerpiece is the Parallel-UNet, which processes information through two distinct streams: one for the "clothing-agnostic" person image (the target person with their original clothes masked out) and another for the "source garment" image.
Instead of calculating exactly where each pixel should move, garment features are fused into the person-UNet using a cross-attention mechanism. This allows the model to establish long-range correspondences, effectively "warping" the garment to the target pose implicitly within the feature space. Unlike previous sequential methods, warping and blending happen simultaneously in a single network pass. To reach 1024×1024 resolution, the authors use a cascaded approach: a base Parallel-UNet generates a 128×128 image, followed by a 256×256 Parallel-UNet super-resolution (SR) step, and finally a standard 1024×1024 SR diffusion network.
Data and Training Insights
The model was trained on a massive dataset of 4 million image pairs, where each pair consists of the same person wearing the same garment in two different poses. This scale allowed the model to generalize across a vast diversity of body shapes, skin tones, and complex garment types like ruffles and intricate patterns.
A key insight is the use of attention over flow. Explicit flow estimation is a bottleneck; cross-attention handles "discontinuous" warping—such as when part of a sleeve is hidden behind the back—which traditional flow-based methods struggle to represent. Additionally, for efficiency, the "garment-UNet" is early-stopped as its primary role is providing features for the "person-UNet" to attend to.
Experimental Results
TryOnDiffusion significantly outperformed existing baselines like TryOnGAN, SDAFN, and HR-VITON both quantitatively and qualitatively. In a head-to-head user study, non-expert raters chose TryOnDiffusion as the best result 92.72% of the time on random samples and 95.80% of the time on challenging poses.
Quantitatively, the model achieved the lowest (best) Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) scores on both an internal 6K test set and the VITON-HD benchmark. Visual results show remarkable preservation of text and complex patterns even under challenging pose changes.
Limitations and Future Work
Despite its success, the model has limitations. Preprocessing errors in human parsing can lead to "leaking" artifacts. The "clothing-agnostic" representation may also remove identifying features like tattoos. Furthermore, the model was primarily trained on clean backgrounds and is currently limited to upper-body clothing. Future work aims to extend this to full-body try-on and video sequences.
Summary updated