TL;DR
This paper introduces R&R, a novel query rewriting method for Retrieval-Augmented Generation (RAG) systems that significantly improves performance in specialized domains. By continually pre-training query rewriters on domain-specific documents, R&R equips them with the necessary knowledge to bridge the gap between user queries and technical documents, leading to more accurate answers.
What Problem Does This Paper Address?
Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by retrieving relevant documents to inform their answers. A crucial step in RAG is query rewriting, where user queries are rephrased to better match document content. However, in specialized fields like finance or law, generic query rewriters struggle because they lack the domain-specific knowledge needed to understand the nuances of technical language and jargon. This "Query-Document Discrepancy" leads to poor retrieval and, consequently, inaccurate answers. The paper aims to solve this by equipping query rewriters with domain expertise.
Methods
The core of the R&R method is Continual Pre-Training (CPT). This involves taking a pre-trained LLM and further training it on a corpus of domain-specific documents. The training objective is standard next-token prediction, similar to how students might review textbooks to prepare for an open-book exam. This process imbues the rewriter with domain knowledge.
Additionally, the authors explore Supervised Fine-Tuning (SFT) after CPT. Since creating domain-specific query rewriting datasets is expensive, they propose a cost-effective method using advanced LLMs (like GPT-4) to automatically generate synthetic training data by analyzing question-answer pairs and identifying key rewrites. The combination of CPT and SFT is referred to as the R&R method.
Datasets
The paper evaluates R&R on several datasets:
- Professional QA Datasets:
- SRCQA: A newly created dataset of 2,742 multiple-choice questions from the Chinese Sponsor Representative Competency Examination, covering finance, accounting, and regulatory knowledge.
- SyllabusQA: A dataset for course logistics.
- FintextQA: A dataset for broad finance.
- Lawbench-Issue: A subset of the Lawbench dataset focusing on legal issue identification.
- General QA Dataset:
- AmbigQA: A dataset designed to test query ambiguity.
- Supplementary Datasets:
- PubMedQA: A biomedical question-answering dataset.
Key Insights and Ideas
- Domain Knowledge is Crucial for Rewriting: The paper strongly emphasizes that generic query rewriters are insufficient for specialized domains. The ability to understand domain-specific terminology and concepts is paramount for effective query rewriting.
- CPT is More Beneficial for Rewriting than Direct Answering: While CPT helps models acquire domain knowledge, this knowledge is more effectively leveraged by the query rewriting component of RAG rather than for direct question answering by the LLM reader.
- R&R is Resource-Efficient: The CPT process is surprisingly fast, with training a 7B model on 100k tokens taking only about 44 seconds on a single NVIDIA 4090 GPU.
- CPT and SFT are Complementary: Ablation studies show that while both CPT and SFT improve performance individually, their combination yields the best results, suggesting they address different aspects of the rewriting task (domain knowledge vs. task execution).
Key Results
- State-of-the-Art Performance in Professional QA: R&R significantly outperforms baseline methods on professional QA datasets, particularly SRCQA, achieving an accuracy of 0.622 with a Qwen2.5-7B model. This is a substantial improvement over other methods.
- Effective Query-Document Discrepancy Reduction: The R&R method effectively reduces the semantic gap between user queries and documents, with CPT further enhancing this reduction.
- Maintained General QA Performance: While optimized for professional QA, R&R does not negatively impact performance on general QA tasks.
- Expert-Level Performance on SRCQA: The model achieved an accuracy of over 0.6 on the challenging SRCQA dataset, indicating near-expert performance in this specialized domain.
- Efficiency with Various LLMs: R&R demonstrates versatility by performing well with different base LLMs, including smaller models like Gemma2-2B.
Limitations and Future Work
- Limited Improvement in General Domain: The method's gains are less pronounced in general domains compared to specialized ones.
- Dependency on Domain Corpora: The effectiveness of R&R relies on the availability of domain-specific corpora for CPT, posing a challenge in document-scarce scenarios.
- Handling Complex Reasoning: The current R&R approach shows limitations in handling queries that require multi-hop reasoning or complex logical analysis. Future work will focus on integrating R&R with reasoning retrieval models to address these challenges.
Summary updated