CF-TCIR: A Compositor-Free Framework for Hierarchical Text-Conditioned Image Retrieval

Yuchen Yang, Yu Wang, Yanfeng WangACL 2024 · OpenReview

Abstract

In text-conditioned image retrieval (TCIR), the combination of a reference image and modification text forms a query tuple, aiming to locate the most congruent target image within a dataset. While traditional techniques often employ attention-driven compositors to craft a unified image-text representation, our paper introduces a compositor-free framework, CF-TCIR, which eschews the standard compositor. We reformulate the retrieval process as a cross-modal interaction between a synthesized image feature and its corresponding text descriptor. This methodology offers advantages in terms of computational efficiency, scalability, and superior performance. We advocate a tiered retrieval mechanism, blending both coarse-grain and fine-grain paradigms, and integrate a generative cross-modal alignment technique using GPT-2 to ensure synchronization of sequential attributes.

Introduction

Text-Conditioned Image Retrieval (TCIR) is a critical task in computer vision where a user seeks a target image based on a reference image and a natural language modification text (e.g., "Find me something like this shirt, but long-sleeved and lighter gray"). Traditionally, this has been approached by building complex "compositors"—heavy neural modules designed to fuse image and text features. However, the CF-TCIR framework presented at ACL 2024 suggests that these complex fusion layers may be over-engineered and prone to overfitting.

The Problem with Traditional Compositors

Existing TCIR models typically rely on attention-driven compositors to create a joint representation of the query. The authors identify two primary limitations:

  1. Overfitting: Complex compositors often memorize dataset-specific quirks instead of learning generalizable attribute correlations, hindering performance on out-of-distribution data.
  2. Attention Contradiction: Standard multi-modal attention focuses on congruence (similarity), whereas TCIR is fundamentally about identifying differences (the "delta") between the reference and target images.
Comparison between traditional compositor-based frameworks and the proposed hierarchical CF-TCIR framework.

The CF-TCIR Framework

CF-TCIR introduces three key innovations to address these challenges:

1. Compositor-Free Fusion

Instead of a learned fusion network, the model employs a non-parametric operation that calculates a "reversed channel attention" map. By suppressing feature channels in candidate images that are too similar to the reference image, the model is forced to focus on the channels representing the requested textual modifications.

2. Hierarchical Retrieval

To improve efficiency, CF-TCIR uses a two-stage pipeline:

  • Coarse-grained Retrieval: The modification text alone is used to filter out the vast majority of irrelevant database entries.
  • Fine-grained Retrieval: Detailed image-text fusion is performed only on the top remaining candidates, significantly boosting speed without compromising accuracy.

3. Generative Cross-modal Alignment

To better synchronize visual and textual attributes, the authors integrate a frozen GPT-2 model during training. By mapping image features into a sequence space, they use a generative loss to ensure that sequential attributes (e.g., "long-sleeved" followed by "lighter gray") are correctly aligned across modalities.

Experimental Results

The model was evaluated on three major fashion benchmarks: FashionIQ, Shoes, and Fashion200k. CF-TCIR consistently outperformed or matched state-of-the-art models that use more computationally expensive encoders.

Performance comparison on the FashionIQ dataset across various categories.

On the FashionIQ dataset, CF-TCIR achieved an Average Recall@10 of 35.00, surpassing previous benchmarks like CLIP4Cir. On the Shoes dataset, it set a new record with 18.95% Recall@1. Furthermore, the hierarchical strategy allowed the model to maintain high accuracy while performing detailed fusion on only the top 5% of candidate images.

Conclusion

CF-TCIR demonstrates that a "less is more" approach can be highly effective for TCIR. By replacing complex compositors with a lightweight, non-parametric fusion and a hierarchical search strategy, the framework achieves superior efficiency and performance. Future work may involve integrating even more powerful foundation models or scaling the hierarchical strategy to billion-scale datasets.

Check out the full paper: CF-TCIR: A Compositor-Free Framework for Hierarchical Text-Conditioned Image Retrieval (ACL 2024)

Summary updated

Abstract

In text-conditioned image retrieval (TCIR), the combination of a reference image and modification text forms a query tuple, aiming to locate the most congruent target image within a dataset. The advantages of rich image semantic information and text flexibility are combined in this manner for more accurate retrieval. While traditional techniques often employ attention-driven compositors to craft a unified image-text representation, our paper introduces a compositor-free framework, CF-TCIR, which eschews the standard compositor. Compositor-based methods are designed to learn a joint representation of images and text, but they struggle to directly capture the correlations between attributes across the image and text modalities. Instead, we reformulate the retrieval process as a cross-modal interaction between a synthesized image feature and its corresponding text descriptor. This novel methodology offers advantages in terms of computational efficiency, scalability, and superior performance. To optimize the retrieval performance, we advocate a tiered retrieval mechanism, blending both coarse-grain and fine-grain paradigms. Moreover, to enrich the contextual relationship within the query tuple, we integrate a generative cross-modal alignment technique, ensuring synchronization of sequential attributes between image and text data.
BibTeX
@inproceedings{yang2024cf,
  author = {Yuchen Yang and Yu Wang and Yanfeng Wang},
  title = {{CF-TCIR: A Compositor-Free Framework for Hierarchical Text-Conditioned Image Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=kFk7ZG9DyM},
}

Loading related content...

Figure