Optimizing LLM Based Retrieval Augmented Generation Pipelines in the Financial Domain

Yiyun Zhao, Prateek Singh, Hanoz Bhathena, Bernardo Ramos, Aviral Joshi, Swaroop Gadiyaram, Saket SharmaNAACL 2024 · DOI

TL;DR

This paper dives deep into optimizing Retrieval Augmented Generation (RAG) systems for the financial domain. By systematically evaluating different LLMs, retrieval strategies, and prompt designs, they reveal crucial insights into what makes RAG effective, particularly highlighting the sensitivity of LLMs to retrieval quality and the significant performance gap between leading proprietary models and open-source alternatives.

What Problem Does This Paper Address?

Large Language Models (LLMs) are powerful, but they often struggle with domain-specific knowledge and up-to-date information, leading to "hallucinations" – generating factually incorrect or fabricated content. This unreliability is a major hurdle for their adoption in critical areas like finance. Retrieval Augmented Generation (RAG) aims to solve this by grounding LLM responses in external knowledge sources. However, there's a lack of systematic understanding of how each component of a RAG pipeline (retriever, prompts, LLM) impacts the final answer quality, especially in a highly specialized domain like finance. This paper addresses this gap by providing a comprehensive evaluation framework to understand these interactions and offer practical recommendations for building better RAG systems.

Methods

The authors designed a rigorous evaluation framework to benchmark RAG pipelines. They simulated different retrieval qualities by manipulating the presence, order, and number of retrieved documents. Six LLMs (GPT-4, GPT-3.5-turbo, Llama-2-7B, Llama-2-13B, and their chat variants) were tested across 15 retrieval scenarios and 9 different prompt configurations. Answer quality was primarily measured using Token F1 scores, and instruction following ability was also assessed. A baseline extractive model (RoBERTa) was used for comparison.

Datasets

The study utilized two custom datasets curated from the financial (banking) sector:

  • Banking webpages: Publicly available information on banking products.
  • Banking policy guides: Internal guides for customer service executives.

These datasets were populated with real user queries and paired with relevant documents. The authors emphasize that these datasets were used purely for evaluation and not for fine-tuning the models.

Key Insights and Ideas

  • LLMs are "pseudo-helpful" but sensitive to retrieval: Even advanced LLMs like GPT-4 tend to generate answers even when relevant information is missing, a phenomenon termed "pseudo-helpfulness." This means they might provide related but incorrect information or hallucinate, rather than stating "No answer found." This highlights that simply maximizing retrieval recall isn't enough; precision and the quality of retrieved documents are paramount.
  • Retrieval quality is king: The performance of LLMs in RAG systems is highly dependent on the quality of the retrieved documents. Distractor documents or poorly ranked relevant documents significantly degrade answer quality, even for top-tier models.
  • Proprietary models lead the pack: OpenAI's GPT-4 and GPT-3.5-turbo consistently outperformed open-source Llama-2 models across various metrics, showing greater robustness to prompt variations and better instruction following.
  • Prompting has less impact than expected on top models: While prompts can influence LLM behavior, GPT-4 and GPT-3.5-turbo demonstrated remarkable resilience to prompt variations, suggesting they are less susceptible to prompt engineering nuances compared to Llama-2 models.
  • Order matters: The position of the "gold" document in the retrieved list significantly impacts answer quality, reinforcing the need for re-ranking mechanisms in retrieval optimization.

Key Results

  • Generative LLMs significantly outperform baseline extractive models in answer quality.
  • GPT-4 showed superior performance over GPT-3.5-turbo, which in turn outperformed the Llama-2 models.
  • Placing the correct document at the top of the retrieved list led to better performance.
  • GPT-4 and GPT-3.5-turbo achieved over 94% accuracy in following structured output instructions (pipe and JSON formats), while Llama-2 models struggled, especially with the "pipe" format.
  • GPT models also demonstrated higher accuracy in quoting directly from sources compared to Llama-2 models.

Limitations and Future Work

The authors acknowledge several limitations:

  • The inherent challenge of comprehensively identifying all relevant documents for a query in RAG systems.
  • The assumption that answers are derived from a single passage, which may not hold true for all real-world scenarios.
  • The study was limited to six models from two families (OpenAI GPT and Llama); evaluating a wider range of models (e.g., Claude, Mistral) and datasets is suggested.
  • Retrieval quality was simulated through manipulation, rather than using real-world retrieval systems. Future work could explore how improvements at the retrieval stage translate to enhanced answer quality.

They suggest developing algorithms for more precise document identification and broadening the scope to more complex applications.

Summary updated

BibTeX
@inproceedings{zhao2024optimizing,
  author = {Yiyun Zhao and Prateek Singh and Hanoz Bhathena and Bernardo Ramos and Aviral Joshi and Swaroop Gadiyaram and Saket Sharma},
  title = {{Optimizing LLM Based Retrieval Augmented Generation Pipelines in the Financial Domain}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-industry.23},
}
Figure