On the Robustness of Generative Information Retrieval Models

Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Changjiang Zhou, Maarten de Rijke, Xueqi Cheng

Introduction

Generative Information Retrieval (GIR) represents a paradigm shift in search technology. While traditional "index-retrieve" pipelines rely on matching query vectors to document vectors in a pre-built index, GIR models—typically based on Transformer architectures—are trained to directly generate a unique document identifier (docid) in response to a query. While these models have achieved state-of-the-art results on standard benchmarks, their reliability in real-world, "out-of-distribution" (OOD) scenarios remains under-explored. This paper presents a systematic stress test of GIR models, investigating whether their performance stems from true generalization or mere memorization.

Methodology and Evaluation Framework

To evaluate OOD robustness, the authors propose a taxonomy covering four critical perspectives:

  1. Query Variations: Testing resilience against misspellings, word reordering, and paraphrasing.
  2. Unseen Query Types: Assessing performance on complex, multi-hop questions when trained on simpler web search queries.
  3. Unseen Tasks: Evaluating cross-task generalization (e.g., from fact-checking to entity linking).
  4. Corpus Expansion: Measuring performance degradation as new documents are added post-training.

The study utilizes the KILT (Knowledge-Intensive Language Tasks) benchmark, encompassing 11 datasets across tasks like Fact Checking (FEVER), Open Domain QA (NQ, HotpotQA, TriviaQA), Slot Filling (T-REx), Entity Linking (AY2), and Dialogue (WoW).

Examples of query variations used to test model robustness, including misspellings and structural changes.

The researchers compared two GIR models (BART and CorpusBrain) against traditional Sparse (BM25) and Dense (DPR) retrieval models. Robustness was quantified using a "Drop Rate" ($DR_{OOD}$) metric, measuring the performance delta between standard and OOD settings.

Key Insights: The Strengths and Fragilities of GIR

The systematic evaluation revealed several critical characteristics of generative models:

  • The Autoregressive Achilles' Heel: GIR models are highly sensitive to query quality. Because they generate docids character-by-character, a single typo can derail the entire retrieval process.
  • The "Bag-of-Words" Advantage: The classic BM25 algorithm remains remarkably robust against query perturbations. Its lack of sensitivity to word order or exact syntax allows it to handle "messy" real-world queries better than sophisticated neural counterparts.
  • Pre-training as a Shield: Specialized pre-training, as seen in CorpusBrain, significantly bolsters OOD robustness. Training tasks that emphasize the relationship between document segments help the model generalize beyond its immediate training data.
  • The GIR Superpower in Corpus Expansion: Surprisingly, GIR models excel at handling growing document collections. By using prefix trees to store indexes and beam search for retrieval, they maintain better performance than dense retrieval models (DPR) as the search space expands.

Comparative Performance Results

The experimental results highlight the trade-offs between different retrieval paradigms. While GIR models often lead in raw performance on standard tasks, their "Drop Rate" in OOD scenarios can be significant.

Performance comparison of various retrieval models across different KILT tasks.

Key findings include:

  • Misspelling Sensitivity: In the FEVER dataset's misspelling scenario, BART's performance plummeted by 74%, compared to a 54% drop for DPR and only 36% for BM25.
  • Task Generalization: GIR models showed superior robustness in "Slot Filling" tasks, likely because the structured nature of these tasks aligns well with the generative docid format.
  • Scalability: When the corpus was expanded by 40%, CorpusBrain maintained higher R-precision than DPR, particularly in Fact Checking tasks.

Conclusion and Future Directions

This research characterizes GIR models as "strong but brittle." While they offer elegant architectures and high peak performance, their sensitivity to query noise and task shifts poses challenges for production environments. Future work should explore alternative docid formats—such as semantic numeric strings—and develop more diverse benchmarks to test the limits of generative retrieval, including resistance to adversarial attacks. This study serves as a foundational roadmap for developing more reliable and robust generative search engines.

Summary updated

Abstract

Generative information retrieval methods retrieve documents by directly generating their identifiers. Much effort has been devoted to developing effective generative IR models. Less attention has been paid to the robustness of these models. It is critical to assess the out-of-distribution (OOD) generalization of generative IR models, i.e., how would such models generalize to new distributions? To answer this question, we focus on OOD scenarios from four perspectives in retrieval problems: (i)query variations; (ii)unseen query types; (iii)unseen tasks; and (iv)corpus expansion. Based on this taxonomy, we conduct empirical studies to analyze the OOD robustness of representative generative IR models against dense retrieval models. Our empirical results indicate that the OOD robustness of generative IR models is in need of improvement. By inspecting the OOD robustness of generative IR models we aim to contribute to the development of more reliable IR models. The code is available at \url{https://github.com/Davion-Liu/GR_OOD}.
BibTeX
@inproceedings{liu2025robustness,
  author = {Yu-An Liu and Ruqing Zhang and Jiafeng Guo and Changjiang Zhou and Maarten de Rijke and Xueqi Cheng},
  title = {{On the Robustness of Generative Information Retrieval Models}},
  booktitle = {ECIR},
  year = {2025},
  doi = {10.1007/978-3-031-88711-6\_26},
}
Figure