When evaluating how well a search engine understands human language, we face a fundamental paradox: the benchmarks we use to test semantic understanding are often deeply rooted in surface-level keyword matching.
In information retrieval (IR), the transition from traditional sparse models (like BM25) to dense neural retrievers (like bi-encoders) promised a shift from literal word matching to deep semantic understanding. However, evaluating these models objectively has proven difficult. A growing body of research reveals that our standard evaluation datasets suffer from a systemic lexical evaluation bias. Because of how these benchmarks are constructed, they inadvertently penalize neural models that capture meaning, while artificially inflating the performance of systems that rely on exact keyword overlap.
The Mechanism of the Blind Spot
To understand why evaluation is biased, we must look at how IR test collections are built. Historically, creating a dataset required a technique called pooling. Since human annotators cannot read millions of documents for every search query, researchers run a set of baseline search engines and pool the top results for human review.
As Emine Yilmaz et al. demonstrated in On the Reliability of Test Collections for Evaluating Systems of Different Types, when these pools are populated primarily by traditional keyword-matching systems, they fail to capture the relevant documents that semantic models are uniquely equipped to find. If a dense retriever finds a highly relevant document that uses paraphrases or synonyms instead of the exact query terms, that document is frequently marked as "irrelevant" simply because it was never included in the historical keyword-generated pool.
[Search Query]
│
├─► Sparse Baselines (BM25) ──► Pool of Documents ──► Human Labeling ──► Ground Truth
│ ▲
└─► Dense Retriever ──────────► Unique Semantic Document ───────────────────┘
(Marked IRRELEVANT because it missed the pool)
This issue is compounded by the tools used during data annotation. Ji Ma et al. noted in Hybrid First-stage Retrieval Models for Biomedical Literature that human annotators frequently use term-based search engines to locate ground-truth documents in the first place. This dependency creates a closed feedback loop: the dataset favors keyword-heavy models because the ground truth itself was discovered via keywords.
Furthermore, this bias extends to how humans judge relevance. In QUADRo: Dataset and Models for QUestion-Answer Database Retrieval, Stefano Campese et al. observed that human labeling processes frequently suffer from a cognitive shortcut where annotators prioritize similar wording over actual semantic equivalence.
The Illusion of Sparse Superiority
This structural flaw severely distorts our understanding of model capabilities, particularly when testing how well models generalize to new domains without explicit training data.
In A Thorough Examination on Zero-shot Dense Retrieval, Ruiyang Ren et al. showed that the perceived superiority of keyword-based retrieval in zero-shot settings is frequently an artifact of benchmark construction. When the authors evaluated dense models on datasets where target documents were not selected based on exact word matches, the dense retrieval models became highly competitive.
Without accounting for this, standard metrics may exhibit a mathematical bias toward term-based models, as argued by Yuefeng Li et al. in Mining Specific and General Features in Both Positive and Negative Relevance Feedback. Similarly, Jia-Huei Ju et al. concluded in CFDA & CLIP Labs at TREC'23 Product Search Track that systematic bias in relevance judgments leads to a consistent underestimation of dense vector-based models.
Even modern attempts to scale evaluation using Large Language Models (LLMs) as automated judges run into this trap. In Revisiting Human-vs-LLM judgments using the TREC Podcast Track, Watheq Mansour et al. found that automated evaluation frameworks often inherit or amplify a preference for lexical keyword matching, risking penalizing neural search architectures that capture meaning rather than exact word overlaps.
How Lexical Bias Corrupts Model Behavior
The consequences of lexical bias are not confined to evaluation metrics; they actively degrade how models are trained. Because neural models are optimized to perform well on these biased benchmarks, they learn pathological shortcuts.
Overfitting to Token Overlap in RAG
In Retrieval-Augmented Generation (RAG) systems, retrievers are tasked with finding evidence for downstream generation. However, Mirella Lapata et al. demonstrated in Retrieval Augmented Generation with Rich Answer Encoding that standard models often overfit to passages with high token overlap with the target answer, ignoring factually correct evidence that uses different wording.
The Collapse into Literal Matching
When dense models are forced to maximize performance on lexically biased datasets, they can experience what Ali Modarressi et al. term a "collapse." In Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence, the authors showed that dense retrievers frequently abandon semantic reasoning entirely, prioritizing literal keyword matches over semantically equivalent synonyms. This makes them highly vulnerable to keyword-stuffing tactics and causes them to miss critical contextual nuances.
The Inverse Problem: The Lexical Gap
While lexical bias unfairly penalizes dense models for finding synonyms, it highlights a legitimate weakness in neural architectures: their inability to handle highly specific, rare keywords when precision is paramount.
As Thibault Formal et al. explored in Match Your Words! A Study of Lexical Matching in Neural Information Retrieval, dense bi-encoders systematically underestimate the importance of rare or highly discriminative terms compared to human relevance judgments. Because they map queries into a continuous semantic space, they can smooth over a unique serial number, product ID, or rare medical term in favor of broad contextual similarity.
This tension shows why the field cannot simply abandon lexical matching. Instead, researchers are developing hybrid frameworks. For instance, Dohyeon Lee et al. proposed the Ratio of Complementarity (RoC) metric in On Complementarity Objectives for Hybrid Retrieval to explicitly force sparse and dense models to solve different parts of the search problem, rather than allowing the lexical bias of the dataset to dictate the behavior of both.
Designing Resilient Evaluation
To move past the limitations of lexical bias, the information retrieval field is shifting toward evaluation frameworks that decouple relevance from exact term matching. This involves:
- Constructing Unbiased Pools: Building evaluation datasets where ground-truth documents are discovered using a diverse ensemble of dense, sparse, and generative retrieval mechanisms to eliminate the sparse-pooling blind spot.
- Semantic Annotation: Designing human and automated labeling protocols that explicitly reward semantic equivalence and factual support rather than surface-level token overlap.
- Hybrid Objectives: Training models using objectives that preserve the dense model's capacity for deep semantic abstraction while maintaining the exact-match precision required for out-of-domain generalization.
Key Foundational Work
Latent Retrieval for Weakly Supervised Open Domain Question Answering (2019) — contributed 1 concept elements to this direction.
Automated Fact Checking in the News Room (2019) — contributed 1 concept elements to this direction.
Mining Specific and General Features in Both Positive and Negative Relevance Feedback (2009) — contributed 1 concept elements to this direction.