Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All

Mehmet Saygin Seyfioglu, Karim Bouyarmane, Suren Kumar, Amir Tavanaei, Ismail B. TutararXiv 2024 · DOI · arXiv

Introduction

The e-commerce industry is rapidly moving toward highly personalized shopping experiences, yet the technical challenge of Virtual Try-All (Vit-All) remains significant. Unlike traditional virtual try-on, which is typically limited to apparel, Vit-All aims to integrate any product—from furniture and home decor to accessories—into any user-provided scene.

Existing solutions generally fall into two categories, both with significant drawbacks. Zero-shot models (e.g., Paint-by-Example) offer real-time performance but often lose fine-grained details like textures, logos, and specific patterns due to the "information bottleneck" of global image embeddings. Conversely, personalization models (e.g., DreamPaint) preserve these details through fine-tuning but require prohibitive computation time (often ~40 minutes per product), making them impractical for large-scale retail. Diffuse to Choose (DTC) bridges this gap, providing high-fidelity, zero-shot Vit-All in seconds.

Methodology: The DTC Architecture

DTC is built upon a Latent Diffusion Model (Stable Diffusion v1.5) and introduces several architectural innovations to maintain product identity without the need for per-item fine-tuning.

DTC Architecture

The core innovation is the Hint Signal. Instead of relying solely on a global semantic vector, DTC creates a spatial "hint" by stitching a resized version of the reference product into the masked area of the source image. This signal is processed by a secondary U-Net encoder—a trainable replica of the main encoder—to extract granular, pixel-level spatial features.

To integrate these features into the main diffusion process, the authors use Feature-wise Linear Modulation (FiLM). This method affinely aligns feature maps between the hint encoder and the main U-Net decoder, which the authors found more effective for detail preservation than standard cross-attention. Additionally, DTC replaces the standard CLIP image encoder with DINOv2 for richer semantic embeddings and utilizes a VGG-based perceptual loss during training to ensure color and texture consistency.

Results and Key Insights

DTC demonstrates that pixel-level conditioning is essential for maintaining product identity. By re-introducing the reference image through a secondary pathway, the model avoids the information loss typical of global CLIP tokens or the VAE latent space bottleneck.

Qualitative Vit-All Results

Quantitative Performance

On the VITON-HD-NoFace benchmark, DTC significantly outperformed existing zero-shot baselines:

  • CLIP Score: 90.14 (vs. 87.79 for Paint-by-Example)
  • FID: 5.39 (vs. 5.93 for Paint-by-Example)

Speed and User Preference

DTC performs inference in approximately 6 seconds on an NVIDIA A100 GPU, making it viable for production environments. In user studies, DTC's zero-shot outputs were rated nearly as high in similarity and semantic blending as those from DreamPaint, despite the latter requiring significant per-item optimization time.

Conclusion

Diffuse to Choose effectively solves the "information bottleneck" in image-conditioned inpainting. By leveraging a secondary U-Net to process pixel-level hints and integrating them via FiLM, DTC enables high-quality Virtual Try-All at production-ready speeds. Future research will focus on improving the rendering of intricate text and handling complex human poses to further enhance the realism of the generated scenes.

Summary updated

Abstract

As online shopping is growing, the ability for buyers to virtually visualize products in their settings-a phenomenon we define as "Virtual Try-All"-has become crucial. Recent diffusion models inherently contain a world model, rendering them suitable for this task within an inpainting context. However, traditional image-conditioned diffusion models often fail to capture the fine-grained details of products. In contrast, personalization-driven models such as DreamPaint are good at preserving the item's details but they are not optimized for real-time applications. We present "Diffuse to Choose," a novel diffusion-based image-conditioned inpainting model that efficiently balances fast inference with the retention of high-fidelity details in a given reference item while ensuring accurate semantic manipulations in the given scene content. Our approach is based on incorporating fine-grained features from the reference image directly into the latent feature maps of the main diffusion model, alongside with a perceptual loss to further preserve the reference item's details. We conduct extensive testing on both in-house and publicly available datasets, and show that Diffuse to Choose is superior to existing zero-shot diffusion inpainting methods as well as few-shot diffusion personalization algorithms like DreamPaint.
BibTeX
@misc{seyfioglu2024diffuse,
  author = {Mehmet Saygin Seyfioglu and Karim Bouyarmane and Suren Kumar and Amir Tavanaei and Ismail B. Tutar},
  title = {{Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All}},
  year = {2024},
  eprint = {2401.13795},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.13795},
}
Figure