Introduction
Retrieval-Augmented Generation (RAG) has become a cornerstone for grounding Large Language Models (LLMs) in external knowledge. However, RAG systems are not immune to hallucinations, particularly when retrieved documents are irrelevant or insufficient to answer a user's query. A robust RAG model should exhibit two key behaviors: "rejecting incapables" (refusing to answer when information is missing) and "answering capables" (providing accurate responses when data is present). Achieving this balance requires high-quality preference datasets for fine-tuning, but the cost of human annotation for thousands of conversation logs is often prohibitive.
The AL4RAG Framework
To address the high cost of data labeling, we introduce AL4RAG, an active learning framework designed to select the most informative RAG conversation logs for human annotation. Traditional active learning metrics often fail in RAG contexts because the length of retrieved documents tends to overshadow the short user queries in standard similarity calculations.
Retrieval-Augmented Similarity (ras)
The core innovation of AL4RAG is the Retrieval-Augmented Similarity (ras) metric. Unlike standard metrics that treat the prompt as a single text block, ras evaluates the query and the reference documents independently. By decoupling these components, the framework ensures that diversity in user intent is preserved and not masked by repetitive or similar reference texts. Empirical results showed that TF-IDF vectorization for ras outperformed deep semantic embeddings, likely due to its sensitivity to structural and lexical variations in RAG prompts.
Preference Dataset Construction and DPO
Once informative samples are selected, they are annotated for hallucinations. We construct a preference dataset by pairing model responses with canned refusals. If a model's original response is hallucinated, the refusal ("Sorry, this question is beyond my ability") is designated as the "chosen" response. Conversely, if the model answers correctly, its response is preferred over the refusal. The LLM is then optimized using Direct Preference Optimization (DPO) on this curated set.
Experimental Results
Evaluations on the RAGTruth dataset demonstrate that AL4RAG significantly improves model performance while reducing annotation effort.
Superior Efficiency and Performance
Key findings include:
- High Rejection Accuracy: AL4RAG achieved a rejection rate of 22.65% using only 12.5% of the available data, outperforming entropy-based and random sampling baselines.
- Stability on Answerable Queries: The model maintains high performance on valid queries. At a 25% data budget, AL4RAG reached a ROUGE-L score of 22.12, surpassing models trained on larger, less curated datasets.
- Data Quality over Quantity: Fine-tuning with DPO on a 25% subset selected by AL4RAG yielded better results than using 100% of the data. This suggests that the framework effectively filters out noise and redundancy that can degrade preference learning.
Conclusion and Future Work
AL4RAG provides a cost-effective pathway for aligning RAG models with human expectations of honesty and accuracy. By intelligently selecting conversation logs, the framework enables models to better distinguish between what they know and what they do not. Future research will focus on generating more nuanced, context-aware refusals and exploring hybrid vectorization methods to further refine the ras metric.
Summary updated