PromptDresser: Improving the Quality and Controllability of Virtual Try-On via Generative Textual Prompt and Prompt-aware Mask

Jeongho Kim, Hoiyeong Jin, Sunghyun Park, Jaegul Choo

Abstract

Virtual try-on (VTO) technology has transitioned from simple 2D overlays to sophisticated diffusion-based generation. However, existing systems often struggle with stylistic controllability, such as changing how a garment is worn (e.g., tucked vs. untucked). PromptDresser addresses these limitations by introducing a framework that combines image-based clothing transfer with generative text prompts. By leveraging Large Multimodal Models (LMMs) for detailed captioning and a novel prompt-aware masking technique, PromptDresser enables highly realistic and customizable fashion previews that break the structural constraints of original garments.

Introduction

While current VTO tools are proficient at swapping garments, they frequently fail to modify the wearing style. If a source image features a tucked-in shirt, the generated output typically inherits this configuration. This lack of flexibility stems from a reliance on purely image-driven processes and "clothing-agnostic" masks that inadvertently preserve the silhouette of the original clothing. PromptDresser, developed by researchers at KAIST, introduces text-based control to overcome these hurdles, allowing users to specify styles like "loose fit" or "rolled sleeves" through natural language.

Methodology

PromptDresser's architecture is built upon Stable Diffusion XL (SDXL) and incorporates three primary innovations to enhance controllability and quality:

  1. LMM-driven Disentangled Captioning: Utilizing GPT-4o, the system generates independent descriptions for the person (pose, body shape) and the garment (material, neckline). This disentanglement prevents "textual conflict," ensuring the description of the original clothing does not interfere with the generation of the new item.
  2. Random Dilation Mask Augmentation: During training, the model is exposed to masks that are randomly expanded. This teaches the U-Net to generate clothing even when the mask does not perfectly align with the garment's initial shape, reducing reliance on the original silhouette.
  3. Prompt-aware Mask Generation (PMG): This coarse-to-fine approach is the key to inference-time flexibility. Instead of a static mask, PromptDresser performs a partial denoising step to predict the new clothing's location based on the text prompt. It then refines the inpainting mask to match this predicted shape, allowing for structural changes like untucking a shirt.

Key Insights: Breaking the Mask Constraint

A significant finding of this research is that text descriptions can effectively replace complex pose-estimation networks, avoiding the artifacts often introduced by specialized sub-networks. More importantly, the realization that the traditional agnostic mask acted as a bottleneck led to the development of PMG. By allowing the mask to evolve during generation, the model gains the freedom to alter the structural fit of the clothes.

Figure 2: A floral top styled as tucked-in, demonstrating standard alignment.
Figure 3: The same floral top styled as untucked, showcasing the flexibility enabled by prompt-aware masking.

Experimental Results

PromptDresser was evaluated on major benchmarks including VITON-HD, DressCode, and SHHQ-1.0. It achieved state-of-the-art performance across several metrics:

  • Quantitative Performance: On VITON-HD, PromptDresser reached an FID of 8.54, outperforming competitors like IDM-VTON (9.20) and StableVITON (9.84).
  • Style Accuracy: In text-alignment tests, the model correctly executed an "untucked" style 89.42% of the time, nearly doubling the accuracy of baseline models (approx. 46-50%).
  • User Preference: A study with 40 participants showed an 84% preference for PromptDresser's text-based editing capabilities over previous best-in-class models.

Conclusion

PromptDresser successfully integrates natural language control into the virtual try-on pipeline, significantly improving both quality and stylistic flexibility. While currently limited by the 77-token constraint of the CLIP text encoder, future iterations may leverage more advanced models like Stable Diffusion 3 to handle even more complex and granular fashion prompts. This work paves the way for more interactive and personalized digital fashion experiences.

Summary updated

BibTeX
@misc{kim2024promptdresser,
  author = {Jeongho Kim and Hoiyeong Jin and Sunghyun Park and Jaegul Choo},
  title = {{PromptDresser: Improving the Quality and Controllability of Virtual Try-On via Generative Textual Prompt and Prompt-aware Mask}},
  year = {2024},
  eprint = {2412.16978},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.16978},
}
Figure