ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval

Nicola Messina, Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Fabrizio Falchi, Giuseppe Amato, Rita Cucchiara

Abstract

ALADIN (ALign And DIstill Network) addresses the critical efficiency-effectiveness trade-off in image-text retrieval. By distilling fine-grained alignment scores from a complex "early-fusion" teacher into a fast "common-space" student model, ALADIN achieves performance competitive with state-of-the-art Transformers while operating nearly 90 times faster. This approach bridges the gap between slow, high-accuracy cross-modal interaction and fast, lower-accuracy vector-based retrieval.

Introduction

In cross-modal retrieval—such as searching for an image using a text query—researchers typically face a frustrating trade-off. "Early-fusion" models (like Oscar or UNITER) feed images and text into a massive Transformer together, allowing for deep interaction. While highly accurate, they are computationally prohibitive at scale. Conversely, "late-fusion" or "common-space" models project modalities into a shared vector space for fast k-Nearest Neighbor (kNN) search, but often suffer a significant drop in accuracy because they fail to capture fine-grained relationships between specific image regions and words.

Methods

The ALADIN architecture utilizes a dual-head approach to provide both speed and accuracy. It uses two instances of a pre-trained Vision-Language (VL) Transformer (VinVL) with shared weights, keeping visual and textual pipelines disentangled so that features can be cached independently.

The ALADIN architecture features a dual-head design: an Alignment Head for fine-grained scoring and a Matching Head for efficient common-space retrieval.
  1. The Alignment Head: This component computes a similarity matrix between all image regions and all words in a sentence, using a "max-over-regions sum-over-words" pooling strategy to produce high-quality relevance scores.
  2. The Matching Head: This head produces simple, fixed-length vectors for efficient kNN search. Crucially, it is trained via knowledge distillation. The Alignment Head acts as a "teacher," using its fine-grained scores to supervise the Matching Head through a learning-to-rank process (ListNet-style loss). This ensures the fast common space matches the ranking distribution produced by the complex alignment head.

Key Results

ALADIN was evaluated on the MS-COCO dataset using standard 1K and 5K test sets. Its performance is best summarized by its position on the efficiency-effectiveness frontier.

Effectiveness vs. Efficiency plot showing ALADIN filling the gap between fast, low-accuracy models and slow, high-accuracy models.
  • Speed: The distilled Matching Head is approximately 90 times faster than the VinVL early-fusion baseline.
  • Accuracy: On the MS-COCO 5K test set, ALADIN (A/ft.
  • D/ft.) achieved an rsum (sum of recall values) of 215.1, remarkably close to the 231.7 achieved by the much slower VinVL.
  • State-of-the-Art Comparison: ALADIN outperformed previous common-space methods like TERN and TERAN by significant margins, proving that distillation is a superior training signal compared to standard triplet losses.

Conclusion

ALADIN demonstrates that fine-grained interaction can be successfully distilled into efficient common spaces, offering a viable path for large-scale, high-accuracy retrieval. While the current reliance on region-based features remains a minor bottleneck, future work exploring grid-based features or end-to-end vision transformers could further streamline the pipeline. Additionally, the architecture supports a two-stage retrieval pipeline where the fast Matching Head finds candidates for the Alignment Head to re-rank.

Summary updated

Abstract

Image-text matching is gaining a leading role among tasks involving the joint understanding of vision and language. In literature, this task is often used as a pre-training objective to forge architectures able to jointly deal with images and texts. Nonetheless, it has a direct downstream application: cross-modal retrieval, which consists in finding images related to a given query text or vice-versa. Solving this task is of critical importance in cross-modal search engines. Many recent methods proposed effective solutions to the image-text matching problem, mostly using recent large vision-language (VL) Transformer networks. However, these models are often computationally expensive, especially at inference time. This prevents their adoption in large-scale cross-modal retrieval scenarios, where results should be provided to the user almost instantaneously. In this paper, we propose to fill in the gap between effectiveness and efficiency by proposing an ALign And DIstill Network (ALADIN). ALADIN first produces high-effective scores by aligning at fine-grained level images and texts. Then, it learns a shared embedding space – where an efficient kNN search can be performed – by distilling the relevance scores obtained from the fine-grained alignments. We obtained remarkable results on MS-COCO, showing that our method can compete with state-of-the-art VL Transformers while being almost 90 times faster. The code for reproducing our results is available at https://github.com/mesnico/ALADIN.
BibTeX
@inproceedings{messina2022aladin,
  author = {Nicola Messina and Matteo Stefanini and Marcella Cornia and Lorenzo Baraldi and Fabrizio Falchi and Giuseppe Amato and Rita Cucchiara},
  title = {{ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval}},
  booktitle = {CBMI},
  year = {2022},
  url = {https://openreview.net/forum?id=yCxz5UmZjr},
}
Figure