TL;DR
This paper tackles the problem of "hallucinations" in diffusion-generated images used for interactive text-to-image retrieval. They introduce Diffusion-aware Multi-view Contrastive Learning (DMCL), a novel training framework that teaches retrieval models to ignore these hallucinated visual cues, leading to significantly improved retrieval accuracy.
What Problem Does This Paper Address?
Interactive Text-to-Image Retrieval (I-TIR) allows users to refine their search through multi-round dialogues. A recent advancement, Diffusion-Augmented Interactive Text-to-Image Retrieval (DAI-TIR), uses diffusion models to generate "proxy" images based on the dialogue context. These generated images are intended to provide richer visual cues than text alone. However, diffusion models can "hallucinate" — introduce visual elements or attributes not present in the original text query. These hallucinations can conflict with the user's true intent, confusing the retrieval system and degrading performance. Existing DAI-TIR methods treat these generated images as just another input, failing to account for these potentially misleading visual details.
Methods
The core of the paper is the Diffusion-aware Multi-view Contrastive Learning (DMCL) framework. DMCL trains a retrieval encoder to be robust against diffusion-induced hallucinations. It achieves this through two main objectives:
- Multi-View Query-Target Alignment: This objective treats the text query, the diffusion-generated image, and a fused representation of both as multiple views of the same user intent. It uses a generalized symmetric contrastive loss with hard-negative mining to align all these views with the ground-truth target image. This encourages the model to focus on consistent semantic signals and suppress spurious ones.
- Text-Diffusion Consistency: This objective explicitly enforces agreement between the text query and its corresponding diffusion-generated image. It uses both feature-level consistency (aligning embeddings) and distribution-level agreement (ensuring similar retrieval score distributions) to reduce "cross-view drift" and prevent the diffusion image from deviating too far from the textual intent.
Essentially, DMCL trains the encoder to act as a "semantic filter," learning to map hallucinated cues into a null space while preserving genuine intent signals.
Datasets
- DA-VisDial: A new, large-scale dataset constructed by reformulating dialogue contexts from the VisDial dataset using an LLM and then generating proxy images with a diffusion model. This dataset is designed to support DAI-TIR research.
- VisDial validation split: Used for in-distribution evaluation.
- Four additional I-TIR datasets: ChatGPT_BLIP2, HUMAN_BLIP2, Flan-Alpaca-XXL_BLIP2, and PlugIR_dataset, used for out-of-distribution (zero-shot) testing.
Key Insights and Ideas
- Hallucinations are a significant problem: The paper empirically demonstrates that a substantial portion (around 40%) of diffusion-generated images contain inconsistencies with the dialogue text, directly impacting retrieval performance.
- DMCL acts as a "semantic filter": The proposed framework doesn't discard diffusion augmentation but rather trains the retrieval encoder to intelligently filter out hallucinated content, making it more robust.
- Multi-view learning is key: By considering text, diffusion images, and fused representations as complementary views, DMCL can leverage the strengths of each while mitigating the weaknesses of individual views.
- Consistency between text and diffusion is crucial: Simply aligning to the target image isn't enough; ensuring the diffusion image aligns with the text query itself is vital for preventing drift.
Key Results
- Consistent improvements: DMCL consistently outperforms baselines across five benchmarks, achieving significant gains in multi-round Hits@10.
- State-of-the-art performance: DMCL achieved up to 7.37% improvement over prior fine-tuned and zero-shot baselines on the VisDial dataset.
- Robustness to out-of-distribution data: DMCL demonstrated strong generalization capabilities on four additional I-TIR datasets, outperforming baselines even when the data distribution differs.
- Qualitative evidence: Attention visualizations show DMCL focusing on relevant subject regions, while DAR (a baseline) exhibits scattered attention. Similarity distributions also show DMCL having tighter, higher scores, indicating better semantic alignment.
Limitations and Future Work
- Query fusion scheme: The paper acknowledges that DMCL currently uses a simple additive fusion scheme for query views. Future work could explore more sophisticated fusion strategies.
- Release of dataset: The authors are releasing the DA-VisDial dataset to facilitate future research in this area.
Summary updated