Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capables

Xuzhao Geng, Haozhao Wang, Jun Wang, Wei Liu, Ruixuan Li

Introduction

Retrieval-Augmented Generation (RAG) has become a cornerstone for grounding Large Language Models (LLMs) in external knowledge. However, RAG systems are not immune to hallucinations, particularly when retrieved documents are irrelevant or insufficient to answer a user's query. A robust RAG model should exhibit two key behaviors: "rejecting incapables" (refusing to answer when information is missing) and "answering capables" (providing accurate responses when data is present). Achieving this balance requires high-quality preference datasets for fine-tuning, but the cost of human annotation for thousands of conversation logs is often prohibitive.

The AL4RAG Framework

To address the high cost of data labeling, we introduce AL4RAG, an active learning framework designed to select the most informative RAG conversation logs for human annotation. Traditional active learning metrics often fail in RAG contexts because the length of retrieved documents tends to overshadow the short user queries in standard similarity calculations.

The AL4RAG framework illustrates the transition from raw RAG conversations to a fine-tuned LLM using active learning and DPO.

Retrieval-Augmented Similarity (ras)

The core innovation of AL4RAG is the Retrieval-Augmented Similarity (ras) metric. Unlike standard metrics that treat the prompt as a single text block, ras evaluates the query and the reference documents independently. By decoupling these components, the framework ensures that diversity in user intent is preserved and not masked by repetitive or similar reference texts. Empirical results showed that TF-IDF vectorization for ras outperformed deep semantic embeddings, likely due to its sensitivity to structural and lexical variations in RAG prompts.

Preference Dataset Construction and DPO

Once informative samples are selected, they are annotated for hallucinations. We construct a preference dataset by pairing model responses with canned refusals. If a model's original response is hallucinated, the refusal ("Sorry, this question is beyond my ability") is designated as the "chosen" response. Conversely, if the model answers correctly, its response is preferred over the refusal. The LLM is then optimized using Direct Preference Optimization (DPO) on this curated set.

Experimental Results

Evaluations on the RAGTruth dataset demonstrate that AL4RAG significantly improves model performance while reducing annotation effort.

AL4RAG (ras) achieves higher rejection rates and better stability (ROUGE-L) compared to query-based or prompt-based sampling across various data proportions.

Superior Efficiency and Performance

Key findings include:

  • High Rejection Accuracy: AL4RAG achieved a rejection rate of 22.65% using only 12.5% of the available data, outperforming entropy-based and random sampling baselines.
  • Stability on Answerable Queries: The model maintains high performance on valid queries. At a 25% data budget, AL4RAG reached a ROUGE-L score of 22.12, surpassing models trained on larger, less curated datasets.
  • Data Quality over Quantity: Fine-tuning with DPO on a 25% subset selected by AL4RAG yielded better results than using 100% of the data. This suggests that the framework effectively filters out noise and redundancy that can degrade preference learning.

Conclusion and Future Work

AL4RAG provides a cost-effective pathway for aligning RAG models with human expectations of honesty and accuracy. By intelligently selecting conversation logs, the framework enables models to better distinguish between what they know and what they do not. Future research will focus on generating more nuanced, context-aware refusals and exploring hybrid vectorization methods to further refine the ras metric.

Summary updated

Abstract

Retrieval-augmented generation (RAG) is a key technique for leveraging external knowledge and enhancing the factual accuracy of large language models (LLMs). However, RAG still faces challenges in ensuring fully reliable responses in all scenarios. To address this, it is essential to identify samples that tend to lead to unreliable outputs or guide LLMs toward factually correct responses, which experts then annotate to develop high-quality datasets for refining LLMs. However, the growing scarcity of such datasets makes their creation challenging. This paper proposes using the vast amount of conversations generated from widespread LLM usage to build these datasets, with the goal of training LLMs to appropriately handle queries outside its capabilities while providing accurate responses to manageable ones. Given the impracticality of having experts annotate all conversation records, we introduce AL4RAG, a framework that uses active learning to select the most suitable conversation samples for annotation, thereby optimizing model performance within a limited annotation budget. Additionally, recognizing that traditional active learning methods are not fully compatible with RAG due to unsuitable distance metrics, we develop a novel sample distance measurement for RAG active learning. Extensive experiments show that our method consistently outperforms baselines across multiple metrics.
BibTeX
@misc{geng2025enhancing,
  author = {Xuzhao Geng and Haozhao Wang and Jun Wang and Wei Liu and Ruixuan Li},
  title = {{Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capables}},
  year = {2025},
  eprint = {2502.09073},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.09073},
}
Figure