Abstract
ALADIN (ALign And DIstill Network) addresses the critical efficiency-effectiveness trade-off in image-text retrieval. By distilling fine-grained alignment scores from a complex "early-fusion" teacher into a fast "common-space" student model, ALADIN achieves performance competitive with state-of-the-art Transformers while operating nearly 90 times faster. This approach bridges the gap between slow, high-accuracy cross-modal interaction and fast, lower-accuracy vector-based retrieval.
Introduction
In cross-modal retrieval—such as searching for an image using a text query—researchers typically face a frustrating trade-off. "Early-fusion" models (like Oscar or UNITER) feed images and text into a massive Transformer together, allowing for deep interaction. While highly accurate, they are computationally prohibitive at scale. Conversely, "late-fusion" or "common-space" models project modalities into a shared vector space for fast k-Nearest Neighbor (kNN) search, but often suffer a significant drop in accuracy because they fail to capture fine-grained relationships between specific image regions and words.
Methods
The ALADIN architecture utilizes a dual-head approach to provide both speed and accuracy. It uses two instances of a pre-trained Vision-Language (VL) Transformer (VinVL) with shared weights, keeping visual and textual pipelines disentangled so that features can be cached independently.
- The Alignment Head: This component computes a similarity matrix between all image regions and all words in a sentence, using a "max-over-regions sum-over-words" pooling strategy to produce high-quality relevance scores.
- The Matching Head: This head produces simple, fixed-length vectors for efficient kNN search. Crucially, it is trained via knowledge distillation. The Alignment Head acts as a "teacher," using its fine-grained scores to supervise the Matching Head through a learning-to-rank process (ListNet-style loss). This ensures the fast common space matches the ranking distribution produced by the complex alignment head.
Key Results
ALADIN was evaluated on the MS-COCO dataset using standard 1K and 5K test sets. Its performance is best summarized by its position on the efficiency-effectiveness frontier.
- Speed: The distilled Matching Head is approximately 90 times faster than the VinVL early-fusion baseline.
- Accuracy: On the MS-COCO 5K test set, ALADIN (A/ft.
- D/ft.) achieved an
rsum(sum of recall values) of 215.1, remarkably close to the 231.7 achieved by the much slower VinVL. - State-of-the-Art Comparison: ALADIN outperformed previous common-space methods like TERN and TERAN by significant margins, proving that distillation is a superior training signal compared to standard triplet losses.
Conclusion
ALADIN demonstrates that fine-grained interaction can be successfully distilled into efficient common spaces, offering a viable path for large-scale, high-accuracy retrieval. While the current reliance on region-based features remains a minor bottleneck, future work exploring grid-based features or end-to-end vision transformers could further streamline the pipeline. Additionally, the architecture supports a two-stage retrieval pipeline where the fast Matching Head finds candidates for the Alignment Head to re-rank.
Summary updated