LocalRQA: From Generating Data to Locally Training, Testing, and Deploying Retrieval-Augmented QA Systems

Yu Xiao, Yunan Lu, Yu Zhou2024 · DOI

TL;DR

LocalRQA is an end-to-end, open-source toolkit designed to help researchers and developers build Retrieval-Augmented Generation (RAG) systems from scratch without relying on expensive APIs. By providing curated tools for data generation, model training, and local deployment, it enables 7B-parameter open-source models to match the performance of proprietary giants like GPT-4.

What Problem Does This Paper Address?

Building a Retrieval-Augmented Question-Answering (RQA) system usually falls into two camps. On one side, toolkits like LangChain and LlamaIndex make it incredibly easy to "plug and play" using off-the-shelf models and paid APIs (like OpenAI). On the other side, researchers often have to build custom, fragmented codebases to test new training algorithms or evaluation metrics.

The gap lies in customization and cost. Developers often find it too expensive to rely on paid services at scale, yet they struggle to train smaller, local models that perform just as well. Researchers lack a unified framework to compare new retrieval and generation algorithms against prior work. LocalRQA bridges this gap by offering a modular, "all-in-one" pipeline for the entire RQA lifecycle.

Methods

LocalRQA follows a modular design divided into five stages:

  1. Prepare: Scripts to transform raw documents into training triplets $\langle \text{question, answer, passage} \rangle$ using LLMs, or to reformat existing datasets (like Natural Questions).
  2. Train: This is where LocalRQA shines. It implements advanced research techniques for both:
  3. Retrievers: Including contrastive learning (CTL) and distillation from larger models (DCA/RPG).
  4. Generators: Supporting Supervised Fine-Tuning (SFT) and Fusion-in-Decoder (FiD) approaches.
  5. Assemble: A modular system to "stitch" together retrievers, generators, and custom components (like safety filters) in just a few lines of code.
  6. Evaluate: Built-in support for standard metrics (Recall@k, ROUGE) and "LLM-as-a-judge" (GPT-4 Eval).
  7. Deploy: Integration with high-performance inference engines like vLLM and SGLang, along with Gradio-based UIs for human evaluation and interactive chatting.

Datasets

The authors evaluated the toolkit using:

  • Standard Benchmarks: Natural Questions (NQ), TriviaQA, and MS-Marco.
  • New Showcases: They introduced two new RQA datasets by scraping online documentation from Databricks and Faire. These represent "real-world" use cases where a company needs a QA system for its specific, proprietary technical docs.
  • Synthetic Data: They used LLMs to generate 1,185 training pairs for Databricks and 575 for Faire from raw text passages.

Key Insights and Ideas

  • The Power of 7B: One of the most significant "aha moments" is that a 7B-parameter model (like Mistral or Starling), when fine-tuned correctly using LocalRQA, can perform on par with GPT-4-turbo for domain-specific tasks.
  • Retrieval is the Bottleneck: The paper highlights that even the best generator will fail if the retriever doesn't find the right document. They found that Contrastive Learning (CTL) was the most effective method for boosting retriever performance.
  • Human-in-the-Loop: Unlike many toolkits that focus solely on automated metrics, LocalRQA emphasizes human evaluation. It provides a ready-made UI to collect feedback on "helpfulness" and "correctness," which is vital for the final stages of model deployment.

Key Results

The authors compared their locally trained models against OpenAI’s text-ada-002 (retriever) and GPT-4-turbo (generator).

  • Performance Parity: On the Faire dataset, a combination of a BGE retriever and a Starling-7B generator achieved a 90% human accuracy score, actually outperforming the GPT-4 baseline (88%).
  • Retrieval Accuracy: Locally trained retrievers (using CTL) reached a Recall@4 of ~77% on Databricks, matching the performance of OpenAI’s proprietary embedding models.
  • Efficiency: By integrating acceleration frameworks like FAISS and vLLM, the toolkit ensures that these local models are not just accurate, but fast enough for real-time interactive use.

Limitations and Future Work

  • Hardware Constraints: All experiments were performed on a single A100 (80G) GPU. While this is "local" for a lab, it still represents a significant hardware requirement for individual developers.
  • Model Scaling: The study focused on models up to 7B parameters. The authors suggest that testing larger models (e.g., 70B) could further close the gap with proprietary systems but would require more compute.
  • Algorithm Coverage: While the toolkit includes many state-of-the-art algorithms, the field of RAG is moving fast. The authors commit to adding more training and testing methods as the research evolves.

Summary updated

Abstract

Retrieval-augmented question-answering systems combine retrieval techniques with large language models to provide answers that are more accurate and informative.Many existing toolkits allow users to quickly build such systems using off-the-shelf models, but they fall short in supporting researchers and developers to customize the model training, testing, and deployment process.We propose LOCALRQA 1 , an open-source toolkit that features a wide selection of model training algorithms, evaluation methods, and deployment tools curated from the latest research.As a showcase, we build QA systems using online documentation obtained from Databricks and Faire's websites.We find 7B-models trained and deployed using LOCAL-RQA reach a similar performance compared to using OpenAI's text-ada-002 and GPT-4-turbo.
BibTeX
@misc{xiao2024localrqa,
  author = {Yu Xiao and Yunan Lu and Yu Zhou},
  title = {{LocalRQA: From Generating Data to Locally Training, Testing, and Deploying Retrieval-Augmented QA Systems}},
  year = {2024},
  doi = {10.18653/v1/2024.acl-demos.14},
}
Figure