Lexical Bias in Information Retrieval Evaluation

This direction is split into other directions: lexical evaluation bias. Its last version stays readable and citable here.

Version 1 of 150 papers81 conceptsfrozen 2026-10-04

In the transition from traditional keyword-based search to modern neural retrieval, the field of Information Retrieval (IR) has encountered a persistent invisible hurdle: lexical bias. This phenomenon occurs when evaluation benchmarks and training datasets systematically favor exact word matching over semantic understanding, creating a "lexical loop" that can stifle the progress of neural search models.

The Lexical Loop in Benchmarking

Most established IR benchmarks were constructed using a technique called "pooling." To build a dataset like the classic TREC collections, researchers would run several keyword-based search engines (like BM25) and have humans label only the top results from those specific engines.

As Nandan Thakur and Andreas Rücklé demonstrate in their work on the BEIR benchmark, this creates a systemic blind spot. If a modern neural model identifies a document that is semantically perfect but contains none of the query’s exact keywords, that document was likely never seen by the original human annotators. Consequently, the model is penalized for finding "irrelevant" information, simply because that information was missing from the original lexical-based ground truth.

Ofer Egozi, Evgeniy Gabrilovich, and Shaul Markovitch argue in their AAAI 2008 paper that this discrepancy leads to a significant underestimation of semantic search effectiveness. When evaluation pools are biased toward keyword-matching systems, the very innovation semantic search provides—finding relevant content without literal overlap—is treated as an error.

Why Dense Retrievers "Collapse"

The problem isn't just in how we measure models, but in how they learn. Dense retrievers (bi-encoders) are often trained on these same lexically biased datasets. This can lead to what Ali Modarressi and Mohsen Fayyaz call the "collapse" of dense retrievers.

Instead of learning deep semantic relationships, these models often adopt "heuristic shortcuts." They begin to prioritize:

  1. Literal Keyword Matches: Behaving like a vector-based version of BM25.
  2. Extractive Bias: Favoring documents that contain the exact phrasing of the query, a common artifact in datasets where queries are generated directly from target documents.
  3. Surface-level Features: Prioritizing document length or the position of keywords over factual evidence.

Thibault Formal and colleagues found that neural models often systematically underestimate the importance of rare terms compared to human judgments. This "lexical gap" means that while neural models are better at broad context, they often fail at the precise, high-impact keyword matching that traditional systems excel at, leading researchers to rely on hybrid models to bridge the gap.

Circular Evaluation and Automated Judges

The rise of Large Language Models (LLMs) as automated judges has introduced a new layer of complexity. Watheq Mansour, J. Shane Culpepper, and Joel Mackenzie have noted that automated evaluators may mirror the same lexical biases as the models they are judging.

This creates a "circular evaluation" risk. As explored by Jens-Joris Decorte and others in the context of expert finding systems, if we use an automated system to suggest labels for human validation, the resulting benchmark will inherently favor models that share the same underlying logic as the suggestion engine. This reinforces a preference for literal term overlap and penalizes semantic retrieval methods that do not rely on literal overlaps.

Breaking the Bias

To move beyond these limitations, researchers are developing new training objectives and datasets that force models to look past the surface:

  • Complementarity Objectives: Dohyeon Lee and Seung-won Hwang proposed the Ratio of Complementarity (RoC) to ensure that dense and sparse retrievers solve different parts of the search problem rather than just overlapping on keywords.
  • Rich Answer Encoding: Mirella Lapata and Wenyu Huang introduced objectives like Retriever as Answer Classifier (RAC) to train retrievers to find semantically relevant knowledge even when the wording is lexically distinct from the target answer.
  • Natural Query Datasets: Projects like FAVIQ by Jungsoo Park and colleagues derive claims from natural search logs rather than manual edits. This removes "telltale" lexical cues, forcing models to rely on semantic reasoning.

By acknowledging that our "gold standards" are often tinted by the keyword-matching tools of the past, the field is beginning to build evaluation frameworks that can truly measure—and therefore incentivize—the deep semantic understanding that neural retrieval promises.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_lexical_bias_in_information_retrieval_ev_13a67739,
  title = {Lexical Bias in Information Retrieval Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lexical-bias-in-information-retrieval-evaluation/rdr_13a67739e32e4db5b19f5d2e616c1339},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_lexical_bias_in_information_retrieval_ev_13a67739_v1,
  title = {Lexical Bias in Information Retrieval Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lexical-bias-in-information-retrieval-evaluation/rdr_13a67739e32e4db5b19f5d2e616c1339/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}