Abstract Dual encoders perform retrieval by encoding documents and queries into dense low-dimensional vectors, scoring each document by its inner product with the query. We investigate the capacity of this architecture relative to sparse bag-of-words models and attentional neural networks. Using both theoretical and empirical analysis, we establish connections between the encoding dimension, the margin between gold and lower-ranked documents, and the document length, suggesting limitations in the capacity of fixed-length encodings to support precise retrieval of long documents. Building on these insights, we propose a simple neural model that combines the efficiency of dual encoders with some of the expressiveness of more costly attentional architectures, and explore sparse-dense hybrids to capitalize on the precision of sparse retrieval. These models outperform strong alternatives in large-scale retrieval.
Introduction
Modern information retrieval is characterized by a fundamental trade-off between efficiency and expressivity. Classical sparse models like BM25 are highly efficient and excel at precise keyword matching but lack semantic understanding. Conversely, dense dual encoders (e.g., BERT-based retrievers) capture semantic nuances but often struggle with "fidelity"—the ability to distinguish documents when precise term overlap is required, particularly as document length increases. While cross-attention models offer the highest accuracy by modeling all-to-all interactions between query and document tokens, their computational cost makes them impractical for large-scale first-stage retrieval.
This research investigates the root causes of the performance gap between dense and sparse models on long documents and proposes architectural innovations to bridge this gap without sacrificing efficiency.
The Capacity Bottleneck
A key observation in this study is that fixed-length dense encodings face a "length-capacity bottleneck." As a document grows in length, a single fixed-size vector becomes an increasingly "blurry" representation of its content.
Empirical results on synthetic tasks (like the Containing Passage ICT) show that while sparse models maintain high recall regardless of passage length, the performance of standard dense dual encoders (DE-BERT) degrades significantly as the document length increases. This suggests that a single 768-dimensional vector lacks the capacity to represent long documents with the same fidelity as a sparse bag-of-words model.
Theoretical Fidelity Analysis
To understand this bottleneck, the authors employ the theory of dimensionality reduction and random projections. They prove that the "normalized margin"—the gap between the score of a relevant document and its competitors—shrinks as document length increases.
As shown in the theoretical analysis, to maintain a constant level of retrieval quality (fidelity), the embedding dimension $k$ must grow in proportion to the document length. For a 400-word passage, a dual encoder requires significantly more dimensions than for a 50-word one to achieve the same discriminative power.
Multi-Vector Encodings (ME-BERT)
To address the capacity limitation, the authors propose Multi-Vector Encodings (ME-BERT). Instead of compressing a document into a single vector, ME-BERT represents it as a set of $m$ vectors (for example, the first 8 token embeddings). The relevance score is calculated as the maximum inner product between the query vector and any of the document's $m$ vectors. This approach effectively implements a form of "local" attention, allowing the model to store different aspects of a long document in separate vectors. Crucially, this remains efficient because $m$ vectors can still be indexed using standard Approximate Nearest Neighbor (ANN) search techniques.
Sparse-Dense Hybrids
Recognizing that sparse and dense models have complementary strengths—fidelity and generalization, respectively—the authors explore sparse-dense hybrids. By using a simple linear combination of scores from BM25 and ME-BERT, the resulting ME-HYBRID model achieves the best of both worlds: exact term matching and semantic generalization.
Experimental Results
The models were evaluated on major benchmarks including MS MARCO and Natural Questions (NQ).
- MS MARCO Passage: ME-BERT achieved an MRR@10 of 0.334, a substantial improvement over the standard BERT dual encoder (0.302). The ME-HYBRID model further pushed this to 0.343.
- Document Retrieval: The gains were even more pronounced on the MS MARCO Document task, validating the theory that multi-vector representations are particularly beneficial for longer texts.
- Efficiency: Although ME-BERT increases the index size (3x to 8x), its sub-linear inference time makes it a practical choice for real-world retrieval systems.
Conclusion
This work highlights the fundamental limitations of fixed-length dense representations for long-document retrieval. By introducing multi-vector encodings and sparse-dense hybrids, the authors provide a scalable path toward high-fidelity retrieval that combines the semantic power of neural networks with the precision of classical models. Future work may explore more sophisticated methods for selecting document vectors beyond simple truncation, such as clustering or phrase-based selection.
Summary updated