Abstract
In text-conditioned image retrieval (TCIR), the combination of a reference image and modification text forms a query tuple, aiming to locate the most congruent target image within a dataset. While traditional techniques often employ attention-driven compositors to craft a unified image-text representation, our paper introduces a compositor-free framework, CF-TCIR, which eschews the standard compositor. We reformulate the retrieval process as a cross-modal interaction between a synthesized image feature and its corresponding text descriptor. This methodology offers advantages in terms of computational efficiency, scalability, and superior performance. We advocate a tiered retrieval mechanism, blending both coarse-grain and fine-grain paradigms, and integrate a generative cross-modal alignment technique using GPT-2 to ensure synchronization of sequential attributes.
Introduction
Text-Conditioned Image Retrieval (TCIR) is a critical task in computer vision where a user seeks a target image based on a reference image and a natural language modification text (e.g., "Find me something like this shirt, but long-sleeved and lighter gray"). Traditionally, this has been approached by building complex "compositors"—heavy neural modules designed to fuse image and text features. However, the CF-TCIR framework presented at ACL 2024 suggests that these complex fusion layers may be over-engineered and prone to overfitting.
The Problem with Traditional Compositors
Existing TCIR models typically rely on attention-driven compositors to create a joint representation of the query. The authors identify two primary limitations:
- Overfitting: Complex compositors often memorize dataset-specific quirks instead of learning generalizable attribute correlations, hindering performance on out-of-distribution data.
- Attention Contradiction: Standard multi-modal attention focuses on congruence (similarity), whereas TCIR is fundamentally about identifying differences (the "delta") between the reference and target images.
The CF-TCIR Framework
CF-TCIR introduces three key innovations to address these challenges:
1. Compositor-Free Fusion
Instead of a learned fusion network, the model employs a non-parametric operation that calculates a "reversed channel attention" map. By suppressing feature channels in candidate images that are too similar to the reference image, the model is forced to focus on the channels representing the requested textual modifications.
2. Hierarchical Retrieval
To improve efficiency, CF-TCIR uses a two-stage pipeline:
- Coarse-grained Retrieval: The modification text alone is used to filter out the vast majority of irrelevant database entries.
- Fine-grained Retrieval: Detailed image-text fusion is performed only on the top remaining candidates, significantly boosting speed without compromising accuracy.
3. Generative Cross-modal Alignment
To better synchronize visual and textual attributes, the authors integrate a frozen GPT-2 model during training. By mapping image features into a sequence space, they use a generative loss to ensure that sequential attributes (e.g., "long-sleeved" followed by "lighter gray") are correctly aligned across modalities.
Experimental Results
The model was evaluated on three major fashion benchmarks: FashionIQ, Shoes, and Fashion200k. CF-TCIR consistently outperformed or matched state-of-the-art models that use more computationally expensive encoders.
On the FashionIQ dataset, CF-TCIR achieved an Average Recall@10 of 35.00, surpassing previous benchmarks like CLIP4Cir. On the Shoes dataset, it set a new record with 18.95% Recall@1. Furthermore, the hierarchical strategy allowed the model to maintain high accuracy while performing detailed fusion on only the top 5% of candidate images.
Conclusion
CF-TCIR demonstrates that a "less is more" approach can be highly effective for TCIR. By replacing complex compositors with a lightweight, non-parametric fusion and a hierarchical search strategy, the framework achieves superior efficiency and performance. Future work may involve integrating even more powerful foundation models or scaling the hierarchical strategy to billion-scale datasets.
Check out the full paper: CF-TCIR: A Compositor-Free Framework for Hierarchical Text-Conditioned Image Retrieval (ACL 2024)
Summary updated