LexLIP: Lexicon-Bottlenecked Language-Image Pre-Training for Large-Scale Image-Text Sparse Retrieval

Ziyang Luo, Pu Zhao, Can Xu, Xiubo Geng, Tao Shen, Chongyang Tao, Jing Ma, Qingwei Lin, Daxin Jiang

Abstract

Traditional image-text retrieval (ITR) systems rely on dense vector representations, which face significant scalability challenges in terms of computational cost and memory usage. This paper introduces LexLIP, a novel pre-training framework that shifts the ITR paradigm from dense to sparse retrieval. By forcing models to represent images and text through a "lexicon bottleneck," LexLIP learns to map multimodal data into a discrete, weighted vocabulary space. This enables the use of highly efficient inverted indexes, common in text search engines. Experimental results demonstrate that LexLIP achieves a 5.8x speedup in retrieval and a 19.1x reduction in index storage compared to CLIP, while simultaneously establishing new state-of-the-art performance on standard benchmarks like MSCOCO and Flickr30k.

1. Introduction

Modern image-text retrieval systems typically employ a dual-stream architecture, encoding images and text into high-dimensional dense vectors. Similarity is then calculated using operations like cosine similarity. While effective, this approach requires exhaustive k-nearest neighbor (k-NN) searches across the entire database, which becomes a bottleneck as the number of images scales into the millions or billions. Furthermore, storing these dense vectors requires substantial RAM.

In contrast, traditional text retrieval systems utilize sparse representations and inverted indexes to achieve sub-second search times across massive datasets. The primary challenge in applying this to multimodal tasks is bridging the gap between continuous image data and discrete linguistic tokens. LexLIP addresses this by learning to represent images as a collection of weighted "keywords" from a fixed vocabulary.

Comparison between traditional dense-vector retrieval and the proposed sparse-vector retrieval paradigm.

2. Methodology

2.1 Lexicon-Bottlenecked Architecture

LexLIP utilizes a Vision Transformer (ViT) and a Language Transformer as its backbone encoders. The core innovation is the Lexicon-Bottlenecked Module. Instead of producing a single global embedding, the encoders predict importance weights across a large vocabulary (typically ~30,000 tokens). For instance, an image containing a worker on a roof will be assigned high weights for tokens like "roof," "man," and "work."

2.2 Training Objectives

To ensure these lexicon weights are semantically meaningful, the authors employ a two-phase pre-training strategy:

  1. Lexicon Reconstruction: The model is tasked with reconstructing a masked caption. Crucially, the decoder is "weakened"—it is shallow and aggressively masked—forcing it to rely entirely on a weighted sum of vocabulary embeddings (a Continuous Bag-of-Words, or CBoW) provided by the encoder. This bottleneck ensures the encoder selects the most salient keywords to describe the visual content.
  2. Momentum Lexicon-Contrastive Learning: Building on the first phase, the model uses a momentum-based contrastive learning approach to align the sparse representations of images and text. This further refines the importance weights by contrasting positive image-text pairs against a large queue of negative samples.

3. Experimental Results

3.1 Retrieval Performance

LexLIP was pre-trained on two scales: CC4.3M and CC14.3M. On the Flickr30k (1K test) benchmark, LexLIP achieved a Recall@1 of 78.4 for text-to-image retrieval, surpassing previous state-of-the-art dense models. This indicates that sparse representations do not necessarily trade off accuracy for efficiency.

3.2 Efficiency and Scalability

The true strength of LexLIP is revealed at scale. In a test scenario involving 1 million distractor images, LexLIP demonstrated remarkable efficiency gains over CLIP:

  • Search Speed: LexLIP achieved a retrieval time of approximately 50ms per query, representing a 5.8x speedup over CLIP's 292ms.
  • Storage Efficiency: The sparse index for 1 million images required only 152MB, compared to the 2.9GB required for CLIP's dense index—a 19.1x reduction.
Retrieval time comparison on CPU: LexLIP achieves a 5.8x speedup over CLIP as the candidate pool grows.

3.3 Zero-Shot Robustness

Beyond retrieval, LexLIP shows strong generalization capabilities. It outperformed CLIP on 8 out of 10 zero-shot image classification tasks, including datasets like CIFAR-10 and MNIST. This suggests that the lexicon-based features captured by the bottleneck are robust and semantically rich.

4. Conclusion

LexLIP represents a significant step toward making large-scale multimodal retrieval more practical and accessible. By successfully adapting sparse retrieval techniques to the image-text domain, it offers a solution that is both faster and more memory-efficient than traditional dense models. Future work may explore scaling these sparse indexes to the billion-image level and refining the quantization methods used for lexicon weights.

Summary updated

Abstract

Image-text retrieval (ITR) aims to retrieve images or texts that match a query originating from the other modality. The conventional dense retrieval paradigm relies on encoding images and texts into dense representations with dual-stream encoders. However, this approach is limited by slow retrieval speeds in large-scale scenarios. To address this issue, we propose a novel sparse retrieval paradigm for ITR that exploits sparse representations in the vocabulary space for images and texts. This paradigm enables us to leverage bag-of-words models and efficient inverted indexes, significantly reducing retrieval latency. A critical gap emerges from representing continuous image data in a sparse vocabulary space. To bridge this gap, we introduce a novel pre-training framework, Lexicon-Bottlenecked Language-Image Pre-Training (LexLIP), that learns importance-aware lexicon representations. By using lexicon-bottlenecked modules between the dual-stream encoders and weakened text decoders, we are able to construct continuous bag-of-words bottlenecks and learn lexicon-importance distributions. Upon pre-training with same-scale data, our LexLIP achieves state-of-the-art performance on two ITR benchmarks, MSCOCO and Flickr30k. Furthermore, in large-scale retrieval scenarios, LexLIP outperforms CLIP with 5.8× faster retrieval speed and 19.1× less index storage memory. Beyond this, LexLIP surpasses CLIP across 8 out of 10 zero-shot image classification tasks.
BibTeX
@inproceedings{luo2023lexlip,
  author = {Ziyang Luo and Pu Zhao and Can Xu and Xiubo Geng and Tao Shen and Chongyang Tao and Jing Ma and Qingwei Lin and Daxin Jiang},
  title = {{LexLIP: Lexicon-Bottlenecked Language-Image Pre-Training for Large-Scale Image-Text Sparse Retrieval}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01029},
}
Figure