Introduction
Dense Retrieval (DR) has become the standard architecture for modern search engines and Retrieval-Augmented Generation (RAG) systems. By encoding queries and documents into a shared low-dimensional embedding space, DR models allow for efficient semantic matching. Historically, these models have relied almost exclusively on the Transformer architecture. However, Transformers are limited by the quadratic computational complexity of their self-attention mechanism ($O(L^2)$), which creates a significant bottleneck when processing long-form content such as legal documents, scientific papers, or technical manuals.
This paper introduces the Mamba Retriever, a novel dense retrieval model that replaces the Transformer backbone with the Mamba architecture—a Selective State Space Model (SSM). Mamba offers linear scaling ($O(L)$) with respect to sequence length, promising a solution to the efficiency challenges of long-text retrieval without sacrificing the high-quality representations required for accurate search.
Methodology
The Mamba Retriever is implemented as a bi-encoder model, where both the query and the document are processed independently to produce dense vectors.
Architecture and Embedding Strategy
Unlike Transformers, which utilize global self-attention to relate every token in a sequence to every other token, Mamba utilizes a Selective State Space Model. This mechanism allows the model to compress the input sequence into a fixed-size latent state, effectively "summarizing" the information as it processes the text sequentially.
To generate a single representative embedding for a document, the researchers append a special <EOS> (End of Sequence) token to the input text. The hidden state corresponding to this token at the final layer of the model is extracted as the document's dense representation.
Training Objective
The model is fine-tuned using the InfoNCE loss, a standard contrastive learning objective. This training regime encourages the model to minimize the distance between a query and its relevant document in the embedding space while maximizing the distance from irrelevant "negative" samples.
Experimental Evaluation
The researchers evaluated the Mamba Retriever across three distinct retrieval scenarios to test its versatility and efficiency:
- Short-Text Retrieval: Evaluated on the MS MARCO passage ranking dataset, the industry standard for search relevance.
- Zero-Shot Generalization: Tested using the BEIR benchmark, which consists of 13 diverse tasks ranging from financial Q&A to biomedical research.
- Long-Text Retrieval: Evaluated on LoCoV0, a benchmark featuring documents with average lengths between 22,000 and 58,000 characters.
Key Results and Insights
Effectiveness and Generalization
Despite the lack of a bi-directional attention mechanism, the Mamba Retriever demonstrated state-of-the-art performance. On the MS MARCO benchmark, a Mamba-370M model achieved an MRR@10 of 35.2, outperforming larger Transformer baselines like BERT-large (33.9) and RoBERTa-large (33.9). Furthermore, the model showed superior zero-shot generalization on the BEIR suite compared to similarly sized Transformer models, indicating that the Mamba architecture learns robust, transferable features.
Long-Text Superiority
The linear scaling of Mamba proved particularly effective for long-form retrieval. On the LoCoV0 dataset with an 8k token limit, Mamba-130M achieved an nDCG@10 of 90.7, significantly outperforming specialized Transformer variants like Jina-v2 (85.4) and M2-BERT (85.9). The researchers noted that Mamba’s SSM mechanism is inherently suited for this task; because the model must compress information into a latent state to predict subsequent tokens, it naturally develops a strong "summarization" capability that translates well into document embeddings.
Computational Efficiency
The most striking advantage of the Mamba Retriever is its speed. At a sequence length of 2,000 tokens, Mamba is 4x faster than standard Transformer-based models. As sequence lengths increase to 32,000 tokens, Mamba maintains its efficiency advantage, whereas Transformer models become computationally prohibitive or run out of memory.
Discussion and Future Work
The success of the Mamba Retriever suggests that the "quadratic bottleneck" of Transformers is not a prerequisite for high-quality dense retrieval. However, some limitations remain. The current implementation uses a uni-directional (causal) Mamba model. While this performed well, the authors suggest that a truly bi-directional Mamba architecture could further improve retrieval accuracy.
Additionally, while Mamba's latent state is efficient, it represents a fixed memory capacity. For tasks requiring extremely complex, multi-hop reasoning across massive documents, the lack of global self-attention might eventually pose a challenge. Future research will likely focus on pre-training Mamba and Transformers on identical corpora to further isolate the architectural benefits and exploring hybrid models that combine the strengths of both architectures.
Summary updated