PLTON: Product-Level Try-on with Realistic Clothes Shading and Wrinkles
Lists
Only you see this
Abstract
Image-based virtual try-on systems, which fit new garments onto human portraits, are gaining research attention. An ideal pipeline should preserve the static features of clothes, such as textures and logos, while also generating the adaptive dynamic clothes elements, e.g., shadows and folds. Previous works fail specifically in generating dynamic clothes features, as they maintain the warped in-shop clothes trivially with predicted an alpha mask by composition. To break the dilemma of over-preserving and textures losses, we propose a novel diffusion-based Product-level virtual try-on pipeline, PLTON, which can preserve the fine details of logos and embroideries while producing realistic clothes shading and wrinkles. The main insights are in three folds: 1) Adaptive Dynamic Rendering. Harnessing the potential of a pre-trained diffusion model as a generative prior, we condition it based on image-based visual features and train a novel dynamic feature extractor. This extractor generates dynamic tokens retaining high-fidelity semantic content, enabling the creation of compellingly authentic shadows and wrinkles in virtual garments. 2) Static Characteristics Transformation. PLTON commences by mapping in-shop clothes to the target pose using a conventional warping network. Thereafter, a high-pass filter extracts an High-Frequency Maps (HF-Map) to scrupulously retain static fabric features. These HF-Maps inform the generation of modulation maps via our static extractor, which are subsequently integrated into a fixed U-net for the synthesis of the final, detailed garment images. 3) Retention Enhancement Strategy. We propose a Two-stage hybrid denoising technique to steer the diffusion process towards accurate spatial layouts and color fidelity, thus enhancing retention of visual information. PLTON is fine-tuned solely with a modestly-sized try-on dataset. Extensive quantitative and qualitative experiments conducted on high-resolution datasets substantiate the superiority of our framework in emulating the complex dynamics found in real clothing scenarios.
BibTeX
@inproceedings{zang2024plton,
author = {Yanlong Zang and Han Yang and Jiaxu Miao and Yi Yang},
title = {{PLTON: Product-Level Try-on with Realistic Clothes Shading and Wrinkles}},
booktitle = {IJCNN},
year = {2024},
doi = {10.1109/ijcnn60899.2024.10650070},
}Loading related content...