Abstract
With the development of Generative Adversarial Network, image-based virtual try-on methods have made great progress. However, limited work has explored the task of video-based virtual try-on while it is important in real-world applications. Most existing video-based virtual try-on methods usually require clothing templates and they can only generate blurred and low-resolution results. To address these challenges, we propose a Memory-based Video virtual Try-On Network (MV-TON), which seamlessly transfers desired clothes to a target person without using any clothing templates and generates high-resolution realistic videos. Specifically, MV-TON consists of two modules: 1) a try-on module that transfers the desired clothes from model images to frame images by pose alignment and region-wise replacing of pixels; 2) a memory refinement module that learns to embed the existing generated frames into the latent space as external memory for the following frame generation. Experimental results show the effectiveness of our method in the video virtual try-on task and its superiority over other existing methods.
Introduction
Virtual try-on technology has seen significant success in static images, but transitioning to video introduces two major hurdles: temporal consistency and data availability. Treating each video frame as an independent image often leads to "flickering" and blurred textures as the subject moves. Furthermore, most existing methods rely on "clothing templates"—clean, flat-lay images of a garment—which are not always available in real-world scenarios like social media or e-commerce.
MV-TON (Memory-based Video Virtual Try-on Network) addresses these issues by transferring clothing directly from a model image to a person in a video. By leveraging an external memory mechanism to aggregate information across video frames, the framework produces high-resolution, temporally consistent results that outperform previous state-of-the-art methods in both realism and detail preservation.
Methodology
The MV-TON architecture is split into two specialized stages designed to handle the complexities of video-based transfer.
Try-on Module (Stage I)
This module performs the initial transfer using a 3D appearance flow based on the SMPL human body model. It estimates how pixels from the model image should warp to fit the pose of the person in the video frame. A Region Replacing Block then swaps the clothing and arm pixels while preserving the target person’s face and identity.
Memory Refinement Module (MRM - Stage II)
The MRM is the core innovation of this work. Inspired by memory networks used in video segmentation, this module treats previously generated frames as "external memory." It encodes these frames into Key and Value maps. When generating the current frame, the network "reads" from this memory to retrieve missing details or correct misalignments, ensuring the clothing remains consistent from frame to frame.
Training and Datasets
The researchers utilized the VVT (Video Virtual Try-on) dataset, comprising 791 high-quality videos of fashion models. To train effectively on unpaired data, the authors introduced a Matching Discriminator (MD) that uses correlation maps to ensure the texture and pattern of the generated clothing strictly match the source model image.
Results and Evaluation
MV-TON demonstrated a clear lead over baselines like CP-VTON, M2E-TON, and FW-GAN across both quantitative and qualitative metrics.
Quantitative Performance
On the Cycle Transfer Score (CTS)—a metric proposed by the authors to measure how well a model can "round-trip" a transfer—MV-TON achieved a score of 101.501, significantly outperforming M2E-TON (135.843). A lower transfer cost indicates a more robust model that preserves clothing identity.
Qualitative Analysis and User Study
In a user study with 30 participants, MV-TON was preferred 42.15% of the time, nearly double the preference for the previous video-based state-of-the-art, FW-GAN (21.02%). Visual results show much sharper textures on clothing logos and significantly less "ghosting" or blurring during fast movements.
Conclusion
MV-TON represents a significant step forward in video virtual try-on by eliminating the need for clothing templates and introducing a memory-based refinement process. While the system currently struggles with irregularly shaped clothing silhouettes, the use of space-time aggregation provides a robust foundation for future developments in complex garment layering and dynamic background handling.
Summary updated