FACTS About Building Retrieval Augmented Generation-based Chatbots

Rama Akkiraju, Anbang Xu, Deepak Bora, Tan Yu, Lu An, Vishal Seth, Aaditya Shukla, Pritam Gundecha, Hridhay Mehta, Ashwin Jha, Prithvi Raj, Abhinav Balasubramanian, Murali Maram, Guru Muthusamy, Shivakesh Reddy Annepally, Sidney Knowles, Min Du, Nick Burnett, Sean Javiya, Ashok Marannan, Mamta Kumari, Surbhi Jha, Ethan Dereszenski, Anupam Chakraborty, Subhash Ranjan, Amina Terfai, Anoop Surya, Tracey Mercer, Vinodh Kumar Thanigachalam, Tamar Bar, Sanjana Krishnan, Samy Kilaru, Jasmine Jaksic, Nave Algarici, Jacob Liberman, Joey Conway, Sonu Nayyar, Justin BoitanoarXiv 2024 · OpenReview

TL;DR

This paper introduces the "FACTS" framework (Freshness, Architecture, Cost, Testing, Security) for building robust enterprise-grade Retrieval Augmented Generation (RAG) chatbots. It details 15 critical control points within RAG pipelines and offers practical solutions derived from building three chatbots at NVIDIA, addressing key challenges in enterprise AI deployment.

What Problem Does This Paper Address?

The paper tackles the significant challenge of building effective, secure, and cost-efficient enterprise chatbots powered by generative AI. While Large Language Models (LLMs) and RAG frameworks have revolutionized chatbot capabilities, moving beyond simple dialog flows to sophisticated information retrieval and response generation, their enterprise deployment is far from straightforward. Key hurdles include ensuring data freshness, designing flexible and scalable architectures, managing the significant costs associated with LLMs, rigorously testing these complex systems, and maintaining stringent security and access controls for sensitive enterprise data. The authors aim to provide a structured approach and practical insights to overcome these obstacles.

Methods

The core contribution is the FACTS framework, a mnemonic representing five critical dimensions for successful RAG-based chatbots:

  • Freshness (F): Ensuring the chatbot provides up-to-date information.
  • Architecture (A): Designing flexible and adaptive systems.
  • Cost (C): Managing the economic implications of LLM usage.
  • Testing (T): Implementing comprehensive evaluation strategies.
  • Security (S): Safeguarding sensitive data and access.

The paper also details fifteen control points within RAG pipelines, from data ingestion to monitoring, and proposes specific techniques and remediations for each. They leverage their experience building three internal NVIDIA chatbots (NVInfo, NVHelp, and Scout) to illustrate these concepts. For handling complex queries, they propose an agentic architecture that decomposes queries into sub-queries. They also discuss the trade-offs between using large commercial LLMs and smaller open-source models, including empirical results on accuracy and latency.

Datasets

The paper does not introduce new benchmarks or datasets. Instead, it draws on the internal data sources of NVIDIA for its three case study chatbots:

  • NVInfo Bot: Enterprise internal knowledge (approx. 500M documents, >7 TB) from sources like SharePoint, GoogleDrive, and Slack.
  • NVHelp Bot: Knowledge articles for IT Help and HR benefits (approx. 2K multi-modal documents) from sources like Confluence, ServiceNow, and Jira.
  • Scout Bot: Financial earnings data from public sources (approx. 4K multi-modal documents) including company news, blogs, and SEC filings.

Key Insights and Ideas

  • FACTS Framework is Crucial: The paper emphasizes that a holistic approach considering Freshness, Architecture, Cost, Testing, and Security is essential for enterprise chatbot success, not just focusing on individual RAG components.
  • RAG Pipelines are Complex: There are 15 distinct control points in a RAG pipeline, and optimizing each is vital for overall performance. Neglecting any can lead to significant issues.
  • Hybrid Search is Key: For enterprise data, a combination of lexical (e.g., Elasticsearch) and vector search often yields better retrieval accuracy and coverage than either alone.
  • Agentic Architectures for Complex Queries: Simple RAG is insufficient for multi-part or analytical questions. Agent-based systems capable of query decomposition and orchestration are necessary.
  • LLM Choice Matters for Cost and Performance: Smaller, open-source LLMs are becoming increasingly viable, offering comparable accuracy to larger commercial models with better latency and cost-effectiveness.
  • Security is Paramount: Enterprise chatbots must rigorously adhere to document access controls (ACLs) and implement sensitive data guardrails, especially when dealing with internal corporate data.
  • Testing LLMs as Judges Needs Caution: While "LLM-as-a-judge" can scale testing, it's prone to self-fulfilling prophecies and inherent biases; human oversight remains critical.

Key Results

  • The paper presents empirical results comparing the accuracy and latency of various LLMs (GPT-4, GPT-3.5, Llama-2-70B, Llama-3-70B, Mixtral-8x7B, Mixtral-8x22B) on the NVHelp domain. Llama-3-70B showed strong performance, excelling in answer quality while maintaining acceptable latency.
  • They demonstrate an agent architecture for handling complex queries, illustrated with a Scout bot example answering a multi-part financial earnings question.
  • The paper highlights the importance of metadata enrichment, chunking, and query rephrasing as having the most impact on chatbot response quality.

Limitations and Future Work

  • Human Feedback Integration: While they acknowledge the importance of feedback loops and Reinforcement Learning from Human Feedback (RLHF), their continuous learning pipelines are not yet fully built.
  • LLM Customization: The paper notes that if foundational models don't offer customization, aligning them to human feedback can be difficult.
  • Testing Automation: While they advocate for automated security testing and prompt change testing, the full automation of RAG pipelines for post-production lifecycle management is still a work in progress.
  • Future Work: The authors suggest further research in developing agentic architectures for complex queries, efficiently summarizing large volumes of frequently updated data, incorporating auto-ML for RAG control point optimization, and creating more robust evaluation frameworks for subjective responses and conversations. They also mention ongoing work in collecting high-quality human-annotated data to re-evaluate fine-tuning approaches for embedding models.

Summary updated

Abstract

Enterprise chatbots, powered by generative AI, are emerging as key applications to enhance employee productivity. Retrieval Augmented Generation (RAG), Large Language Models (LLMs), and orchestration frameworks like Langchain and Llamaindex are crucial for building these chatbots. However, creating effective enterprise chatbots is challenging and requires meticulous RAG pipeline engineering. This includes fine-tuning embeddings and LLMs, extracting documents from vector databases, rephrasing queries, reranking results, designing prompts, honoring document access controls, providing concise responses, including references, safeguarding personal information, and building orchestration agents. We present a framework for building RAG-based chatbots based on our experience with three NVIDIA chatbots: for IT/HR benefits, financial earnings, and general content. Our contributions are three-fold: introducing the FACTS framework (Freshness, Architectures, Cost, Testing, Security), presenting fifteen RAG pipeline control points, and providing empirical results on accuracy-latency tradeoffs between large and small LLMs. To the best of our knowledge, this is the first paper of its kind that provides a holistic view of the factors as well as solutions for building secure enterprise-grade chatbots."
BibTeX
@misc{akkiraju2024facts,
  author = {Rama Akkiraju and Anbang Xu and Deepak Bora and Tan Yu and Lu An and Vishal Seth and Aaditya Shukla and Pritam Gundecha and Hridhay Mehta and Ashwin Jha and Prithvi Raj and Abhinav Balasubramanian and Murali Maram and Guru Muthusamy and Shivakesh Reddy Annepally and Sidney Knowles and Min Du and Nick Burnett and Sean Javiya and Ashok Marannan and Mamta Kumari and Surbhi Jha and Ethan Dereszenski and Anupam Chakraborty and Subhash Ranjan and Amina Terfai and Anoop Surya and Tracey Mercer and Vinodh Kumar Thanigachalam and Tamar Bar and Sanjana Krishnan and Samy Kilaru and Jasmine Jaksic and Nave Algarici and Jacob Liberman and Joey Conway and Sonu Nayyar and Justin Boitano},
  title = {{FACTS About Building Retrieval Augmented Generation-based Chatbots}},
  year = {2024},
  eprint = {2407.07858},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.07858},
}
Figure