Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely

Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna Qiu, Lili QiuarXiv 2024 · DOI · arXiv

Abstract

Current Retrieval-Augmented Generation (RAG) research often treats all external data tasks as uniform. This survey introduces a four-level stratification of user queries based on complexity—ranging from simple explicit facts to inferred hidden rationales. By categorizing tasks this way, developers can better select between context-based RAG, specialized small models, and fine-tuning to build more reliable domain-expert applications.

Introduction

Large Language Models (LLMs) frequently encounter "knowledge cutoff" and hallucinations when applied to specialized or private datasets. While RAG and fine-tuning are common solutions, standard pipelines often fail in high-stakes domains like law or medicine. This paper argues that these failures stem from a lack of task decomposition. Asking for a simple date is fundamentally different from asking for a decision based on complex historical patterns. This survey provides a roadmap for defining the reasoning required for diverse data-augmented tasks.

The Four Levels of Query Complexity

The core contribution of this work is the stratification of queries into four distinct levels of increasing difficulty:

  1. Level 1: Explicit Facts: Direct retrieval of information explicitly stated in the text (e.g., "What is the capital of France?").
  2. Level 2: Implicit Facts: Requires connecting multiple facts or applying basic common-sense reasoning (e.g., multi-hop questions).
  3. Level 3: Interpretable Rationales: Following explicit domain-specific rules or workflows provided in the external data (e.g., medical diagnostic guidelines).
  4. Level 4: Hidden Rationales: Inferring implicit expert logic or decision-making patterns from historical examples where the rules are not explicitly documented.
The four levels of query complexity in RAG systems, ranging from explicit facts to hidden rationales.

By identifying the query level first, developers can move away from "plugging leaks" in a generic RAG system and instead build a robust architecture tailored to the specific reasoning depth required.

Benchmarking the Framework

The survey categorizes several well-known benchmarks according to this four-level framework. While many existing datasets focus on Level 1 and Level 2, higher-level tasks often require specialized domain data.

Categorization of popular QA and fact-checking datasets across the proposed complexity levels.
  • Level 1 (Explicit): Includes datasets like NQ (Natural Questions), MS MARCO, and SQuAD.
  • Level 2 (Implicit): Includes multi-hop and reasoning datasets like HotPotQA, StrategyQA, and MuSiQue.
  • Level 3 & 4 (Rationales): These levels typically utilize specialized datasets such as GLOBALCIT for legal reasoning or clinical diagnostic manuals.

Strategic Insights and Technical Solutions

The paper identifies three primary paths for injecting external data:

  • Context: High interpretability but constrained by the model's context window.
  • Small Model Assistance: Using specialized smaller models for routing or error correction.
  • Fine-tuning: High capacity for capturing "hidden" knowledge, though it risks degrading general reasoning capabilities.

For Level 1 and 2 queries, hybrid retrieval (combining BM25 with dense vector search) remains the gold standard. Level 3 tasks benefit significantly from Chain-of-Thought (CoT) prompting and prompt tuning to ensure adherence to provided rules. Level 4 tasks, the most challenging, require In-Context Learning (ICL) with expert examples or Supervised Fine-Tuning (SFT) to capture nuanced "expert intuition."

Conclusion and Future Directions

A key takeaway is that RAG is generally superior for factual updates, while fine-tuning should be reserved for adjusting style and logic. Future research should focus on "routing pipelines" capable of automatically detecting a query's complexity level to trigger the appropriate sub-system. Additionally, improving evaluation metrics for fragmented data synthesis remains a critical challenge for advancing Level 4 reasoning.

Summary updated

Abstract

Large language models (LLMs) augmented with external data have demonstrated remarkable capabilities in completing real-world tasks. Techniques for integrating external data into LLMs, such as Retrieval-Augmented Generation (RAG) and fine-tuning, are gaining increasing attention and widespread application. Nonetheless, the effective deployment of data-augmented LLMs across various specialized fields presents substantial challenges. These challenges encompass a wide range of issues, from retrieving relevant data and accurately interpreting user intent to fully harnessing the reasoning capabilities of LLMs for complex tasks. We believe that there is no one-size-fits-all solution for data-augmented LLM applications. In practice, underperformance often arises from a failure to correctly identify the core focus of a task or because the task inherently requires a blend of multiple capabilities that must be disentangled for better resolution. In this survey, we propose a RAG task categorization method, classifying user queries into four levels based on the type of external data required and primary focus of the task: explicit fact queries, implicit fact queries, interpretable rationale queries, and hidden rationale queries. We define these levels of queries, provide relevant datasets, and summarize the key challenges and most effective techniques for addressing these challenges. Finally, we discuss three main forms of integrating external data into LLMs: context, small model, and fine-tuning, highlighting their respective strengths, limitations, and the types of problems they are suited to solve. This work aims to help readers thoroughly understand and decompose the data requirements and key bottlenecks in building LLM applications, offering solutions to the different challenges and serving as a guide to systematically developing such applications.
BibTeX
@misc{zhao2024retrieval,
  author = {Siyun Zhao and Yuqing Yang and Zilong Wang and Zhiyuan He and Luna Qiu and Lili Qiu},
  title = {{Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely}},
  year = {2024},
  eprint = {2409.14924},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.14924},
}
Figure