Abstract
Traditional image-text retrieval (ITR) systems rely on dense vector representations, which face significant scalability challenges in terms of computational cost and memory usage. This paper introduces LexLIP, a novel pre-training framework that shifts the ITR paradigm from dense to sparse retrieval. By forcing models to represent images and text through a "lexicon bottleneck," LexLIP learns to map multimodal data into a discrete, weighted vocabulary space. This enables the use of highly efficient inverted indexes, common in text search engines. Experimental results demonstrate that LexLIP achieves a 5.8x speedup in retrieval and a 19.1x reduction in index storage compared to CLIP, while simultaneously establishing new state-of-the-art performance on standard benchmarks like MSCOCO and Flickr30k.
1. Introduction
Modern image-text retrieval systems typically employ a dual-stream architecture, encoding images and text into high-dimensional dense vectors. Similarity is then calculated using operations like cosine similarity. While effective, this approach requires exhaustive k-nearest neighbor (k-NN) searches across the entire database, which becomes a bottleneck as the number of images scales into the millions or billions. Furthermore, storing these dense vectors requires substantial RAM.
In contrast, traditional text retrieval systems utilize sparse representations and inverted indexes to achieve sub-second search times across massive datasets. The primary challenge in applying this to multimodal tasks is bridging the gap between continuous image data and discrete linguistic tokens. LexLIP addresses this by learning to represent images as a collection of weighted "keywords" from a fixed vocabulary.
2. Methodology
2.1 Lexicon-Bottlenecked Architecture
LexLIP utilizes a Vision Transformer (ViT) and a Language Transformer as its backbone encoders. The core innovation is the Lexicon-Bottlenecked Module. Instead of producing a single global embedding, the encoders predict importance weights across a large vocabulary (typically ~30,000 tokens). For instance, an image containing a worker on a roof will be assigned high weights for tokens like "roof," "man," and "work."
2.2 Training Objectives
To ensure these lexicon weights are semantically meaningful, the authors employ a two-phase pre-training strategy:
- Lexicon Reconstruction: The model is tasked with reconstructing a masked caption. Crucially, the decoder is "weakened"—it is shallow and aggressively masked—forcing it to rely entirely on a weighted sum of vocabulary embeddings (a Continuous Bag-of-Words, or CBoW) provided by the encoder. This bottleneck ensures the encoder selects the most salient keywords to describe the visual content.
- Momentum Lexicon-Contrastive Learning: Building on the first phase, the model uses a momentum-based contrastive learning approach to align the sparse representations of images and text. This further refines the importance weights by contrasting positive image-text pairs against a large queue of negative samples.
3. Experimental Results
3.1 Retrieval Performance
LexLIP was pre-trained on two scales: CC4.3M and CC14.3M. On the Flickr30k (1K test) benchmark, LexLIP achieved a Recall@1 of 78.4 for text-to-image retrieval, surpassing previous state-of-the-art dense models. This indicates that sparse representations do not necessarily trade off accuracy for efficiency.
3.2 Efficiency and Scalability
The true strength of LexLIP is revealed at scale. In a test scenario involving 1 million distractor images, LexLIP demonstrated remarkable efficiency gains over CLIP:
- Search Speed: LexLIP achieved a retrieval time of approximately 50ms per query, representing a 5.8x speedup over CLIP's 292ms.
- Storage Efficiency: The sparse index for 1 million images required only 152MB, compared to the 2.9GB required for CLIP's dense index—a 19.1x reduction.
3.3 Zero-Shot Robustness
Beyond retrieval, LexLIP shows strong generalization capabilities. It outperformed CLIP on 8 out of 10 zero-shot image classification tasks, including datasets like CIFAR-10 and MNIST. This suggests that the lexicon-based features captured by the bottleneck are robust and semantically rich.
4. Conclusion
LexLIP represents a significant step toward making large-scale multimodal retrieval more practical and accessible. By successfully adapting sparse retrieval techniques to the image-text domain, it offers a solution that is both faster and more memory-efficient than traditional dense models. Future work may explore scaling these sparse indexes to the billion-image level and refining the quantization methods used for lexicon weights.
Summary updated