Introduction
The e-commerce industry is rapidly moving toward highly personalized shopping experiences, yet the technical challenge of Virtual Try-All (Vit-All) remains significant. Unlike traditional virtual try-on, which is typically limited to apparel, Vit-All aims to integrate any product—from furniture and home decor to accessories—into any user-provided scene.
Existing solutions generally fall into two categories, both with significant drawbacks. Zero-shot models (e.g., Paint-by-Example) offer real-time performance but often lose fine-grained details like textures, logos, and specific patterns due to the "information bottleneck" of global image embeddings. Conversely, personalization models (e.g., DreamPaint) preserve these details through fine-tuning but require prohibitive computation time (often ~40 minutes per product), making them impractical for large-scale retail. Diffuse to Choose (DTC) bridges this gap, providing high-fidelity, zero-shot Vit-All in seconds.
Methodology: The DTC Architecture
DTC is built upon a Latent Diffusion Model (Stable Diffusion v1.5) and introduces several architectural innovations to maintain product identity without the need for per-item fine-tuning.
The core innovation is the Hint Signal. Instead of relying solely on a global semantic vector, DTC creates a spatial "hint" by stitching a resized version of the reference product into the masked area of the source image. This signal is processed by a secondary U-Net encoder—a trainable replica of the main encoder—to extract granular, pixel-level spatial features.
To integrate these features into the main diffusion process, the authors use Feature-wise Linear Modulation (FiLM). This method affinely aligns feature maps between the hint encoder and the main U-Net decoder, which the authors found more effective for detail preservation than standard cross-attention. Additionally, DTC replaces the standard CLIP image encoder with DINOv2 for richer semantic embeddings and utilizes a VGG-based perceptual loss during training to ensure color and texture consistency.
Results and Key Insights
DTC demonstrates that pixel-level conditioning is essential for maintaining product identity. By re-introducing the reference image through a secondary pathway, the model avoids the information loss typical of global CLIP tokens or the VAE latent space bottleneck.
Quantitative Performance
On the VITON-HD-NoFace benchmark, DTC significantly outperformed existing zero-shot baselines:
- CLIP Score: 90.14 (vs. 87.79 for Paint-by-Example)
- FID: 5.39 (vs. 5.93 for Paint-by-Example)
Speed and User Preference
DTC performs inference in approximately 6 seconds on an NVIDIA A100 GPU, making it viable for production environments. In user studies, DTC's zero-shot outputs were rated nearly as high in similarity and semantic blending as those from DreamPaint, despite the latter requiring significant per-item optimization time.
Conclusion
Diffuse to Choose effectively solves the "information bottleneck" in image-conditioned inpainting. By leveraging a secondary U-Net to process pixel-level hints and integrating them via FiLM, DTC enables high-quality Virtual Try-All at production-ready speeds. Future research will focus on improving the rendering of intricate text and handling complex human poses to further enhance the realism of the generated scenes.
Summary updated