Abstract
Hybrid retrieval systems aim to combine the precision of keyword matching (sparse retrieval) with the semantic understanding of neural search (dense retrieval). However, current hybrid models often suffer from high redundancy, where both components solve the same "easy" queries while failing on the same "hard" ones. This paper introduces a new objective called the Ratio of Complementarity (RoC) and a two-level orthogonality constraint designed to ensure that sparse and dense models provide unique, non-overlapping contributions. Our framework achieves state-of-the-art performance across major benchmarks, including MS MARCO, Natural Questions, and TREC Robust04.
Introduction
In Information Retrieval (IR), two primary paradigms exist: sparse retrieval (e.g., BM25), which excels at exact keyword matching, and dense retrieval (e.g., DPR), which leverages neural embeddings to capture semantic similarities. Hybrid retrieval systems attempt to unify these strengths by combining their scores.
A critical challenge identified in this work is the "Intersection Trap." Many state-of-the-art hybrid models rely on residual learning, training the dense model to compensate for the sparse model's errors. Paradoxically, this often causes the dense model to mimic the sparse model's behavior, increasing their overlap (intersection) rather than their unique contributions. As shown in Figure 1, even advanced models like CLEAR exhibit significant overlap with sparse retrieval, limiting the overall system's effectiveness on complex queries.
Methodology: Maximizing Complementarity
To address the redundancy issue, we propose a framework to maximize the Ratio of Complementarity (RoC). The objective is to maximize the number of unique documents retrieved by the dense model ($D$) that the sparse model ($S$) missed, while simultaneously minimizing their intersection.
Two-Level Orthogonality
Our approach enforces complementarity through two distinct mechanisms:
- Embedding-level Orthogonality: We introduce a novel loss function ($L_{ortho}$) that mathematically constrains the dense embedding vectors to be orthogonal to the sparse lexical vectors. This forces the dense model to represent features that are fundamentally different from simple word counts.
- Input-level Orthogonality: During training, we employ a "token perturbation" technique. By identifying exact keyword matches between a query and a document and randomly replacing them with a
[MASK]token, we force the dense model to learn relevance based on context and synonyms rather than relying on the same lexical cues used by the sparse model.
At inference time, the final relevance score is calculated as a linear combination of the sparse and dense scores.
Experimental Evaluation
We evaluated our model on three diverse IR benchmarks: MS MARCO-Passage (8.8 million passages), Natural Questions (NQ), and TREC Robust04.
Key Results
The proposed method consistently outperformed existing state-of-the-art models:
- MS MARCO: Achieved an MRR@10 of 38.63, surpassing the previous best (CLEAR + coCondenser) of 37.99.
- Natural Questions: Reached a Recall@100 of 90.13%, a significant improvement over the DPR baseline (85.4%).
- TREC Robust04: Achieved a Mean Average Precision (MAP) of 36.74, demonstrating strong performance on complex, long-form queries.
Discussion and Insights
Our analysis reveals several key advantages of the complementarity-driven approach. By discouraging overlap, the hybrid system becomes more robust to "adversarial" queries where keywords are replaced by synonyms. Furthermore, this approach improves length generalization; the dense model can "delegate" lexical matching to the sparse component, allowing it to focus on semantic signals in longer documents.
However, a significant limitation remains: Annotation Bias. Many existing datasets were curated using keyword-based engines, potentially biasing ground truth labels toward exact matches. Future research should focus on developing labeling processes that reduce this lexical bias to more accurately evaluate semantic retrieval capabilities.
Summary updated