Mitigating Query Drift in Automatic Query Expansion

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2214 papers308 conceptsfrozen 2026-10-04

In information retrieval, the "lexical gap" is a fundamental hurdle: users often describe what they want using different words than the authors of the documents they seek. To bridge this gap, researchers use Automatic Query Expansion (AQE), a technique that adds related terms to a search query to improve recall.

However, expansion is a double-edged sword. While it can help find relevant documents, it frequently introduces "noise"—irrelevant or overly broad terms that shift the search focus away from the user's original intent. This phenomenon, known as query drift (or topic drift), can cause a search for "jaguar speed" to suddenly return results about luxury cars or NFL teams instead of the animal.

The Mechanics of Drift

Query drift occurs when the expansion mechanism fails to respect the specific context of the original query. As Tomek Strzalkowski and Barbara Vauthey observed in Information Retrieval Using Robust Natural Language Processing, most statistically similar terms are "semantically inadmissible." Without rigorous filtering, these terms dilute the specificity of the search.

The problem is often rooted in the asymmetry of relationships. As Jie Wie, Beng Chin Ooi, and colleagues pointed out in Mining Term Association Rules for Global Query Expansion, knowing that term A implies term B does not mean the reverse is true. Expanding a specific technical term into a general category often leads to a significant drop in precision.

Furthermore, Kevyn Collins-Thompson demonstrated in Estimating Robust Query Models with Convex Optimization that expansion is inherently unstable. Most systems rely on "pseudo-relevance feedback," assuming the top results of an initial search are relevant. If those initial results are even slightly off-topic, the expansion terms derived from them will amplify the error, causing the search to "break" and wander into irrelevant regions of the corpus.

Grounding Expansion in World Knowledge

To combat drift, researchers have moved toward "grounding" queries in external, structured knowledge. Instead of relying solely on term co-occurrence within a noisy document collection, systems like MORAG, developed by Ofer Egozi, Evgeniy Gabrilovich, and Shaul Markovitch, map queries to high-level concepts found in Wikipedia.

In Concept-Based Feature Generation and Selection for Information Retrieval, they show that using "world knowledge" helps maintain the thematic integrity of a search, especially for short, ambiguous queries. Similarly, Hui Yang and Tat-Seng Chua proposed modeling questions as "Events" in Structured use of external knowledge for event-based open domain question answering. By structuring expansion around facets like time, location, and action, they prevent the "linear" addition of terms from diluting the query's focus.

Adaptive and Optimized Weighting

Another approach is to treat expansion as an optimization problem rather than a simple addition. ChengXiang Zhai and Yuanhua Lv introduced adaptive machine learning models in A Study of Adaptive Relevance Feedback that predict the optimal balance between the original query and feedback information for every individual search.

This line of work was extended by Michael Bendersky and Donald Metzler in Parameterized concept weighting in verbose queries. Their framework, PQE, learns to prioritize both the explicit words in a query and the "latent" concepts added during expansion, ensuring that the original intent remains the primary driver of the retrieval process.

The LLM Era: Iterative Refinement

Modern research has shifted toward using Large Language Models (LLMs) to manage the expansion process. In the RRR (Rewrite-Retrieve-Rerank) framework, Daman Arora and colleagues at Microsoft Research use retrieved documents to help an LLM "rewrite" the query. Crucially, they found that using the LLM to prune irrelevant documents before rewriting is essential to prevent the search from deviating from the original intent.

Recent work like GuideCQR by Hwanhee Lee and MERLIN by Donghoon Han emphasize making queries "retriever-friendly" through iterative "chats" or feedback loops. These systems attempt to balance the integration of new, clarifying details with the "thematic integrity" of the initial request.

As He Lin and Rui Zhao note in Fishing for Answers, even advanced iterative agents struggle with drift; if an agent explores an irrelevant document subset early on, it may never find the "golden chunks" needed to answer the user's actual question. The ongoing challenge in the field remains building "semantic guardrails" that allow for exploration without losing the original search goal.

Papers in this version

Showing 60 of 214 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 261 papers
  2. Version 1 · 2026-10-04 · 214 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_query_expansion_noise_9f70a942,
  title = {query expansion noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/query-expansion-noise/rdr_9f70a9427eb14217acdace795ea8f4cc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_mitigating_query_drift_in_automatic_quer_9f70a942_v1,
  title = {Mitigating Query Drift in Automatic Query Expansion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-query-drift-in-automatic-query-expansion/rdr_9f70a9427eb14217acdace795ea8f4cc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}