Abstract
Virtual Try-On (VTON) has recently trended toward complex "dual-network" architectures to preserve garment details, but at a heavy cost to computational efficiency. This paper introduces MN-VTON, a single-network approach that utilizes a novel "Modality-specific Normalization" strategy to outperform state-of-the-art dual-network models in both image and video try-on tasks while significantly reducing overhead.
Introduction
Virtual Try-On is the task of realistically "overlaying" a garment onto a person in an image or video. Historically, the industry has navigated a tug-of-war between two paradigms:
- Single-Network: Early models used one generative network. They were efficient but often "washed away" fine-grained textures and logos because the network struggled to balance person-features and garment-features simultaneously.
- Dual-Network: To fix this, recent state-of-the-art (SOTA) methods added a second "ReferenceNet" specifically to extract garment details. While effective at preserving textures, this doubles the computational load, making high-resolution video try-on prohibitively expensive for real-world deployment.
The authors propose a "renaissance" of the single-network paradigm (the "1-2-1" in the title refers to this shift: 1 network $\rightarrow$ 2 networks $\rightarrow$ back to 1 network). They aim to prove that a single network can handle complex multi-modal inputs (text, garment images, and person videos) if the data is processed correctly.
Methodology: MN-VTON
The core innovation is MN-VTON (Modality-specific Normalization Virtual Try-On). Instead of using a separate network to help the main model "remember" what the clothes look like, the intelligence is baked directly into the main network using two key strategies:
Modality-specific Normalization (MN)
The network receives three types of input: Text (descriptions), Garment Images, and Target Images/Videos. The authors found that treating these differently is key. They apply separate normalization layers (specifically AdaLN-zero) to the text features versus the visual features (garment + person). This prevents the "semantic" text data from interfering with the "structural" visual data.
Feature Split and Fusion
Within the attention layers of the network, internal features are split to mimic the behavior of a dual-network system. This allows the model to learn multi-scale garment features and fuse them with the person's pose and identity without needing an entirely separate set of weights.
Diffusion Transformer Backbone
Unlike many VTON models that use a standard UNet, MN-VTON is built on a Diffusion Transformer (DiT) architecture (specifically CogVideo-2B). This makes it naturally better at handling temporal consistency for video tasks.
Experimental Setup
The model was evaluated on four major benchmarks:
- Image VTON: VITONHD (high-resolution top-wear) and DressCode (diverse upper, lower, and full-body dresses).
- Video VTON: VVT (standard benchmark for moving subjects) and VIVID (high-quality video dataset for fine-grained detail and temporal stability).
Key Insights and Results
A major finding is that the Garment Image and the Target Person Image share similar visual distributions. By concatenating and normalizing them together—while keeping the Text modality separate—the network produces much clearer, artifact-free results compared to treating all three independently.
Performance and Efficiency
MN-VTON sets new benchmarks across the board:
- Image Quality: On VITONHD, it achieved an SSIM of 0.8853, outperforming the previous SOTA, CatVTON (0.8704).
- Video Fidelity: In video tasks (VIVID dataset), it achieved a VFID (Video Frechet Inception Distance) of 0.9237, a significant improvement over the baseline VIVID model's 1.1394.
- Efficiency: By eliminating the "ReferenceNet," the authors reduced the number of trainable parameters significantly (e.g., from ~1700M in some dual-network setups to ~860M for their UNet-based variant) while actually improving accuracy metrics.
- Human Preference: In user studies, participants preferred MN-VTON's results 65-68% of the time over competing methods, citing better preservation of logos and smoother motion in videos.
The model also demonstrates "mask-free" potential, generating realistic results even when person masks are missing or inaccurate, a common failure point for traditional models.
Conclusion
While MN-VTON is more efficient than dual-network models, it still relies on large-scale Diffusion Transformers requiring significant GPU resources. Future work may address extreme poses and scaling for 4K resolution. Overall, MN-VTON proves that a well-designed single-network architecture can reclaim the efficiency of early models without sacrificing the high-fidelity detail of modern dual-network systems.
Summary updated