TL;DR
Large language models (LLMs) equipped with Retrieval-Augmented Generation (RAG) still hallucinate when queries fall outside their capabilities. AL4RAG introduces an active learning framework designed specifically for RAG conversation logs, using a custom distance metric (Retrieval-Augmented Similarity, or ras) to select the most informative interactions for human annotation. By transforming single-turn conversation logs into preference pairs and fine-tuning with Direct Preference Optimization (DPO), the model learns to refuse unanswerable queries while remaining faithful on answerable ones—using only a fraction of the annotation budget.
What Problem Does This Paper Address?
While RAG grounds LLM responses in retrieved external documents, models still suffer from output deviations and hallucinations when presented with conflicting, incomplete, or out-of-capability queries. Ideally, a reliable RAG system should do two things simultaneously:
- Refuse to answer queries it cannot reliably resolve with the retrieved context.
- Faithfully answer queries that fall within its capabilities.
Teaching models this behavior requires preference datasets containing appropriate refusal examples. However, existing conversation logs from deployed RAG systems are largely unlabeled and only contain a single model response per prompt (lacking pre-paired chosen/rejected alternatives). Moreover, manually annotating massive conversation logs is economically impractical. Standard active learning (AL) strategies struggle in RAG settings because they treat inputs as monolithic text blocks, ignoring the distinct structural roles and length disparities between user queries and long retrieved passages.
Methods
To overcome these challenges, the authors design AL4RAG, a full-pipeline active learning and preference tuning framework:
-
Retrieval-Augmented Similarity (
ras): Traditional diversity-based AL computes similarity across the entire input prompt. In RAG, lengthy reference passages or prompt templates swamp short user questions, obscuring sample differences. Therasmetric explicitly evaluates queries ($q$), references ($r$), and full prompts ($p$) independently using TF-IDF vectors: $$\text{ras}(x, y) = \min\left( \text{sim}_p(x, y), \; \frac{1}{2}(\text{sim}_q(x, y) + \text{sim}_r(x, y)) \right)$$ Taking the minimum prevents lengthy references from dominating while ensuring samples with similar queries but differing contexts are distinguished. -
Informative Batch Selection: Samples are iteratively selected using an Informative and Diverse Data Selection (IDDS) score, prioritizing unannotated conversations that are diverse relative to already selected data but representative of the remaining pool.
- Single-Response Preference Construction: Instead of generating competing responses, annotators (or LLM proxies) evaluate the factual reliability label ($h \in \{0, 1\}$) of the model's original answer:
- If faithful ($h=0$): Chosen = original answer; Rejected = canned refusal string.
- If unfaithful ($h=1$): Chosen = canned refusal string; Rejected = original answer.
- Direct Preference Optimization (DPO): The model is fine-tuned using DPO on the constructed preference pairs.
[RAG Logs: Query + Reference + Output]
│
(Active Learning / ras)
▼
[Selected Informative Subset]
│
(Reliability Check h ∈ {0,1})
▼
[Preference Pairs: Output vs Refusal]
│
(DPO Fine-Tuning)
▼
[Calibrated, Trustworthy RAG LLM]
Datasets
The authors utilized RAGTruth, a benchmark covering Question Answering, Summarization, and Data-to-Text tasks.
- Because RAGTruth lacked generations from modern models, the authors generated answers using Llama-3-8B-Instruct and Qwen2.5-7B-Instruct.
- Outputs were annotated for faithfulness using DeepSeek-v3 (with a 3-run majority vote, followed by manual review for conflicts), yielding ~3,000 preference pairs per model split 8:1:1 into train/val/test sets.
- Experiments also validated generalization using Llama-2-7B-chat.
Key Insights and Ideas
- Surface features beat semantic embeddings for AL diversity: Counterintuitively, sparse TF-IDF vectorization outperformed modern dense embedding models (Sentence-BERT and Stella-1.5B). Dense encoders project superficially different phrasing into nearby semantic clusters, whereas TF-IDF better isolates fine-grained lexical variations that often dictate RAG factual boundaries.
- Model answers cause interference during selection: Ablation studies revealed that computing similarity over query + reference yields peak performance. Factoring the model's generated answer into the distance metric degraded performance, acting as noise.
- Less data is often better: Models fine-tuned via DPO on 12.5% to 25% of actively selected data outperformed models trained on 100% of the full dataset. Full training datasets introduce noisy, marginal examples that dilute the model's decision boundary for when to refuse.
Key Results
Under identical data budgets, AL4RAG with ras consistently outperformed random sampling, uncertainty-based AL (Entropy, BLEUVar), and standard diversity baselines (Coreset, Naive IDDS, LDCAL):
- Rejection Performance (Llama-3-8B @ 12.5% data):
- AL4RAG$_{\text{ras}}$: 31.37% Rejection Rate (RR) / 35.37 F1.
- Random Selection: 23.04% RR / 28.25 F1.
- Naive IDDS: 25.00% RR / 30.32 F1.
- Answer Stability & Faithfulness (Llama-3-8B @ 12.5% data):
- AL4RAG$_{\text{ras}}$ achieved 75.11% faithfulness (evaluated via Spark X1), outperforming both random selection (74.53%) and prompt-concatenated IDDS (73.87%), alongside higher ROUGE-L (34.15 vs. 33.83).
- These gains remained consistent across base architectures, including Qwen2.5-7B-Instruct (37.83 F1 at 12.5% vs. 32.73 for Random) and Llama-2-7B-chat.
Limitations and Future Work
- Privacy in Conversation Logs: Sourcing training data directly from real-world user logs introduces data privacy concerns and potential leakage of sensitive user details if privacy filtering is absent.
- Synthetic / LLM Annotation Dependency: Budget limitations led the authors to use DeepSeek-v3 as an LLM annotation proxy for the primary labeling pass, which may inherit model-specific evaluation biases despite majority voting.
- Refusal Granularity: The current setup relies on a binary refusal mechanism with template strings rather than teaching the model to formulate nuanced partial answers or request clarifications.
Summary updated