Cross-Corpus Query Expansion with External Knowledge

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 267 papers89 conceptsfrozen 2026-10-04

Information retrieval often fails because of a "vocabulary gap." A user might search for "feline respiratory illness," while the most relevant document simply discusses "sick cats." This problem is magnified in short-text environments like Twitter or medical forums, where queries are brief and the target content is riddled with slang, typos, or highly specific jargon.

Traditionally, systems tried to fix this using Pseudo-Relevance Feedback (PRF): they would take the top few results from a search, assume they were relevant, and extract new keywords from them to "expand" the original query. However, as Donald Metzler and Fernando Diaz argue in Improving the estimation of relevance models using large external corpora, this creates a dangerous feedback loop. If the first few results are noisy or irrelevant—common in small or uncurated datasets—the expanded query "drifts" away from the user's intent, leading to even worse results.

The solution emerging across these 67 papers is Cross-Corpus Query Expansion: instead of looking for help within the noisy target collection, systems "borrow" context from a high-quality, external knowledge source.

Bridging the Lexical Chasm

The core intuition is that an external corpus—like Wikipedia, a news archive, or the entire Web—acts as a semantic bridge. When a query is too short to provide context, the system uses it to "probe" a more descriptive dataset.

For example, in the QCRI at TREC 2013 Microblog Track, Tarek El-Ganainy and colleagues demonstrated that short Twitter queries could be enriched by retrieving snippets from Google search results. These snippets provide formal news titles and professional vocabulary that bridge the gap between a user's informal query and the diverse language of the "Twitter firehose."

This "bridge" works in several specialized ways:

  • Medical Terminology: Yang Song 0010 showed that layman health queries can be expanded using professional MeSH terms found via web search results, translating "heart attack" into "myocardial infarction" to find clinical documents.
  • Social Media Sparsity: To handle the 140-character limit of early microblogs, Runwei Qiang and the PKUICST team enriched tweets by crawling the full text of the embedded URLs within them, effectively using the linked web page as an external expansion source for the tweet itself.
  • Definition Questions: When a user asks "What is X?", Dong-Hong Ji found that using Wikipedia as a "reference set" provides a stable context that simple keyword matching on a news corpus cannot provide.

Why External Sources Provide Stability

The reason this works better than local feedback is statistical stability. Small or noisy datasets (like a collection of blog posts) have "idiosyncratic" language. If you expand a query using a blog post, you might pick up a specific blogger's typos or unique slang.

Wouter Weerkamp and Maarten de Rijke demonstrate in A Generative Blog Post Retrieval Model that Uses Query Expansion based on External Collections that high-quality, edited corpora (like news archives) provide more reliable word associations. By drawing expansion terms from these "clean" sources, the system avoids the topic drift caused by the unedited nature of user-generated content.

Amit Singhal 0001 at AT&T Labs pioneered this "conservative enrichment" approach, showing that a large, stable external dataset prevents the retrieval system from being misled by the biases inherent in smaller collections.

Global vs. Local Expansion

This shift represents a move from Local Expansion (learning from the current search results) to Global Expansion (learning from the world's knowledge).

In Statistical Machine Translation for Query Expansion in Answer Retrieval, Stefan Riezler treats this as a translation problem. By training on massive external pairs of questions and answers, the system learns a "global" map of how concepts relate. This allows the system to expand a query using patterns learned from millions of examples offline, rather than desperately trying to find clues in a handful of potentially irrelevant local results.

Ultimately, cross-corpus expansion works because it decouples understanding the query from searching the collection. By using Wikipedia or the Web to "understand" what a user means, the system can then return to the target collection with a much more powerful and precise set of search terms.

Papers in this version

Showing 60 of 67 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 116 papers
  2. Version 1 · 2026-10-04 · 67 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cross_corpus_query_expansion_a052ed26,
  title = {cross-corpus query expansion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-corpus-query-expansion/rdr_a052ed266dde48d082243089ac9ac7ed},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_corpus_query_expansion_with_extern_a052ed26_v1,
  title = {Cross-Corpus Query Expansion with External Knowledge},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-corpus-query-expansion-with-external-knowledge/rdr_a052ed266dde48d082243089ac9ac7ed/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}