ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval

Nicola Messina, Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Fabrizio Falchi, Giuseppe Amato, Rita Cucchiara

Abstract

ALADIN (ALign And DIstill Network) addresses the critical efficiency-effectiveness trade-off in image-text retrieval. By distilling fine-grained alignment scores from a complex "early-fusion" teacher into a fast "common-space" student model, ALADIN achieves performance competitive with state-of-the-art Transformers while operating nearly 90 times faster. This approach bridges the gap between slow, high-accuracy cross-modal interaction and fast, lower-accuracy vector-based retrieval.

Introduction

In cross-modal retrieval—such as searching for an image using a text query—researchers typically face a frustrating trade-off. "Early-fusion" models (like Oscar or UNITER) feed images and text into a massive Transformer together, allowing for deep interaction. While highly accurate, they are computationally prohibitive at scale. Conversely, "late-fusion" or "common-space" models project modalities into a shared vector space for fast k-Nearest Neighbor (kNN) search, but often suffer a significant drop in accuracy because they fail to capture fine-grained relationships between specific image regions and words.

Methods

The ALADIN architecture utilizes a dual-head approach to provide both speed and accuracy. It uses two instances of a pre-trained Vision-Language (VL) Transformer (VinVL) with shared weights, keeping visual and textual pipelines disentangled so that features can be cached independently.

The ALADIN architecture features a dual-head design: an Alignment Head for fine-grained scoring and a Matching Head for efficient common-space retrieval.
  1. The Alignment Head: This component computes a similarity matrix between all image regions and all words in a sentence, using a "max-over-regions sum-over-words" pooling strategy to produce high-quality relevance scores.
  2. The Matching Head: This head produces simple, fixed-length vectors for efficient kNN search. Crucially, it is trained via knowledge distillation. The Alignment Head acts as a "teacher," using its fine-grained scores to supervise the Matching Head through a learning-to-rank process (ListNet-style loss). This ensures the fast common space matches the ranking distribution produced by the complex alignment head.

Key Results

ALADIN was evaluated on the MS-COCO dataset using standard 1K and 5K test sets. Its performance is best summarized by its position on the efficiency-effectiveness frontier.

Effectiveness vs. Efficiency plot showing ALADIN filling the gap between fast, low-accuracy models and slow, high-accuracy models.
  • Speed: The distilled Matching Head is approximately 90 times faster than the VinVL early-fusion baseline.
  • Accuracy: On the MS-COCO 5K test set, ALADIN (A/ft.
  • D/ft.) achieved an rsum (sum of recall values) of 215.1, remarkably close to the 231.7 achieved by the much slower VinVL.
  • State-of-the-Art Comparison: ALADIN outperformed previous common-space methods like TERN and TERAN by significant margins, proving that distillation is a superior training signal compared to standard triplet losses.

Conclusion

ALADIN demonstrates that fine-grained interaction can be successfully distilled into efficient common spaces, offering a viable path for large-scale, high-accuracy retrieval. While the current reliance on region-based features remains a minor bottleneck, future work exploring grid-based features or end-to-end vision transformers could further streamline the pipeline. Additionally, the architecture supports a two-stage retrieval pipeline where the fast Matching Head finds candidates for the Alignment Head to re-rank.

Summary updated

BibTeX
@inproceedings{messina2022aladin,
  author = {Nicola Messina and Matteo Stefanini and Marcella Cornia and Lorenzo Baraldi and Fabrizio Falchi and Giuseppe Amato and Rita Cucchiara},
  title = {{ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval}},
  booktitle = {CBMI},
  year = {2022},
  url = {https://openreview.net/forum?id=yCxz5UmZjr},
}

Loading related content...

Figure