A Survey on RAG Meets LLMs: Towards Retrieval-Augmented Large Language Models

Yujuan Ding, Wenqi Fan, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, Qing Li

Abstract

As one of the most advanced techniques in AI, Retrieval-Augmented Generation (RAG) offers reliable and up-to-date external knowledge, providing significant advantages for numerous tasks. In the era of AI-Generated Content (AIGC), the capacity of retrieval to provide additional knowledge enables RAG to assist existing generative AI in producing high-quality outputs. While Large Language Models (LLMs) have demonstrated revolutionary abilities in language understanding and generation, they face inherent limitations such as hallucinations and outdated internal knowledge. Retrieval-Augmented Large Language Models (RA-LLMs) have emerged to harness external and authoritative knowledge bases to augment generation quality. This survey comprehensively reviews RA-LLMs across three technical perspectives: architectures, training strategies, and applications.

Introduction

Large Language Models (LLMs) have fundamentally changed the landscape of artificial intelligence, yet their deployment in mission-critical applications is often hampered by two "Achilles' heels": the knowledge cutoff and the tendency to hallucinate. Retrieval-Augmented Generation (RAG) provides a robust solution by grounding these models in external, authoritative data. This survey maps the rapidly evolving RA-LLM ecosystem, offering a systematic framework for understanding how models can transition from "guessing" based on static weights to "researching" from dynamic databases.

The Core Problem: Why RAG?

The primary limitation of vanilla LLMs is that their internal knowledge is frozen at the moment training concludes. They are unaware of events occurring after their cutoff date and often struggle with domain-specific or private information. When an LLM encounters a query for which it lacks data, it frequently generates "hallucinations"—factually incorrect information that is presented with high linguistic confidence.

Comparison of LLMs with and without RAG, illustrating how external knowledge fixes the knowledge cutoff and hallucination problems.

While fine-tuning can mitigate these issues, it is computationally expensive and impractical for data that changes daily. RA-LLMs solve this by querying external databases (such as Wikipedia, medical journals, or private codebases) at inference time, ensuring the model has access to the most current and relevant information.

The RA-LLM Framework

The RA-LLM architecture is generally structured as a pipeline consisting of three main stages: Retrieval, Augmentation, and Generation.

  1. Retrieval: The system identifies and extracts relevant information from an external database. Modern systems have shifted from traditional keyword-based search (like BM25) to "Dense Retrieval," which uses vector embeddings to capture semantic meaning.
  2. Augmentation: The retrieved data is integrated into the model's processing flow. This can happen at the input layer (prompting), intermediate layers (via cross-attention), or the output layer (merging probability distributions).
  3. Generation: The LLM produces a final response that synthesizes its pre-trained knowledge with the newly retrieved context.
The end-to-end workflow of an RA-LLM, showing the stages of retrieval, augmentation, and generation in a medical diagnosis context.

Technical Perspectives and Strategies

The survey categorizes RA-LLM research into several key technical dimensions:

Retrieval Granularity

Techniques vary in the size of the information units they retrieve, ranging from entire documents to fine-grained token-level or entity-level retrieval. Finer granularity can provide more precise context but often increases computational overhead.

Augmentation Strategies

  • Input-layer Augmentation: The most common approach for "black-box" models (like GPT-4), where retrieved text is simply prepended to the user's prompt.
  • Intermediate-layer Augmentation: Requires "white-box" access to the model, using cross-attention mechanisms to bake retrieved information directly into the model's hidden states.
  • Output-layer Augmentation: Merges the model's output distribution with the distribution of the retrieved data to refine the final token selection.

Training Paradigms

  • Training-free: Utilizing prompt engineering to guide the model without weight updates.
  • Independent Training: Training the retriever and the LLM as separate components.
  • Sequential/Joint Training: Optimizing both components together to ensure the retriever finds exactly what the generator needs for the highest quality output.

Key Results and Insights

RA-LLMs consistently outperform vanilla LLMs across various benchmarks. In Open-domain QA, models like Fusion-in-Decoder (FiD) and Atlas have set new state-of-the-art records by effectively processing dozens of retrieved passages. In specialized fields like law and medicine, RAG has been shown to reduce hallucination rates significantly, making AI more viable for professional use.

Furthermore, RA-LLMs offer a path toward greater efficiency. They allow smaller models (e.g., 7B parameters) to compete with much larger models (e.g., 175B) on knowledge-intensive tasks because the smaller models do not need to store every fact within their internal weights.

Limitations and Future Frontiers

Despite its success, the RA-LLM field faces several ongoing challenges:

  • Trustworthiness & Security: Models are susceptible to "retrieval poisoning," where malicious data in the external database can manipulate the model's output.
  • Multi-modal RAG: Expanding retrieval and generation to include images, video, and audio.
  • Efficiency: High-frequency retrieval remains slow. Future work is focused on document compression and speculative decoding to improve real-time performance.
  • Context Window vs. RAG: As LLM context windows grow to millions of tokens, the community is debating whether RAG will remain a necessity or if "prompt stuffing" will become the dominant paradigm.

Conclusion

RA-LLMs represent a significant leap toward more reliable and capable AI. By grounding generative power in authoritative external knowledge, these models overcome the fundamental limitations of static training, paving the way for more trustworthy AI applications across all sectors.

Summary updated

Abstract

As one of the most advanced techniques in AI, Retrieval-Augmented Generation (RAG) can offer reliable and up-to-date external knowledge, providing huge convenience for numerous tasks. Particularly in the era of AI-Generated Content (AIGC), the powerful capacity of retrieval in providing additional knowledge enables RAG to assist existing generative AI in producing high-quality outputs. Recently, Large Language Models (LLMs) have demonstrated revolutionary abilities in language understanding and generation, while still facing inherent limitations, such as hallucinations and out-of-date internal knowledge. Given the powerful abilities of RAG in providing the latest and helpful auxiliary information, Retrieval-Augmented Large Language Models (RA-LLMs) have emerged to harness external and authoritative knowledge bases, rather than solely relying on the model's internal knowledge, to augment the generation quality of LLMs. In this survey, we comprehensively review existing research studies in RA-LLMs, covering three primary technical perspectives: architectures, training strategies, and applications. As the preliminary knowledge, we briefly introduce the foundations and recent advances of LLMs. Then, to illustrate the practical significance of RAG for LLMs, we systematically review mainstream relevant work by their architectures, training strategies, and application areas, detailing specifically the challenges of each and the corresponding capabilities of RA-LLMs. Finally, to deliver deeper insights, we discuss current limitations and several promising directions for future research. Updated information about this survey can be found at https://advanced-recommender-systems.github.io/RAG-Meets-LLMs/
BibTeX
@inproceedings{ding2024survey,
  author = {Yujuan Ding and Wenqi Fan and Liangbo Ning and Shijie Wang and Hengyun Li and Dawei Yin and Tat-Seng Chua and Qing Li},
  title = {{A Survey on RAG Meets LLMs: Towards Retrieval-Augmented Large Language Models}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2024},
  doi = {10.1145/3637528.3671470},
}
Figure