Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capables

Xuzhao Geng, Haozhao Wang, Jun Wang, Wei Liu, Zhang Jinghua, Ruixuan LiSubmission to ICLR 2026 · Withdrawn

TL;DR

Large language models (LLMs) equipped with Retrieval-Augmented Generation (RAG) still hallucinate when queries fall outside their capabilities. AL4RAG introduces an active learning framework designed specifically for RAG conversation logs, using a custom distance metric (Retrieval-Augmented Similarity, or ras) to select the most informative interactions for human annotation. By transforming single-turn conversation logs into preference pairs and fine-tuning with Direct Preference Optimization (DPO), the model learns to refuse unanswerable queries while remaining faithful on answerable ones—using only a fraction of the annotation budget.

What Problem Does This Paper Address?

While RAG grounds LLM responses in retrieved external documents, models still suffer from output deviations and hallucinations when presented with conflicting, incomplete, or out-of-capability queries. Ideally, a reliable RAG system should do two things simultaneously:

  1. Refuse to answer queries it cannot reliably resolve with the retrieved context.
  2. Faithfully answer queries that fall within its capabilities.

Teaching models this behavior requires preference datasets containing appropriate refusal examples. However, existing conversation logs from deployed RAG systems are largely unlabeled and only contain a single model response per prompt (lacking pre-paired chosen/rejected alternatives). Moreover, manually annotating massive conversation logs is economically impractical. Standard active learning (AL) strategies struggle in RAG settings because they treat inputs as monolithic text blocks, ignoring the distinct structural roles and length disparities between user queries and long retrieved passages.

Methods

To overcome these challenges, the authors design AL4RAG, a full-pipeline active learning and preference tuning framework:

  • Retrieval-Augmented Similarity (ras): Traditional diversity-based AL computes similarity across the entire input prompt. In RAG, lengthy reference passages or prompt templates swamp short user questions, obscuring sample differences. The ras metric explicitly evaluates queries ($q$), references ($r$), and full prompts ($p$) independently using TF-IDF vectors: $$\text{ras}(x, y) = \min\left( \text{sim}_p(x, y), \; \frac{1}{2}(\text{sim}_q(x, y) + \text{sim}_r(x, y)) \right)$$ Taking the minimum prevents lengthy references from dominating while ensuring samples with similar queries but differing contexts are distinguished.

  • Informative Batch Selection: Samples are iteratively selected using an Informative and Diverse Data Selection (IDDS) score, prioritizing unannotated conversations that are diverse relative to already selected data but representative of the remaining pool.

  • Single-Response Preference Construction: Instead of generating competing responses, annotators (or LLM proxies) evaluate the factual reliability label ($h \in \{0, 1\}$) of the model's original answer:
  • If faithful ($h=0$): Chosen = original answer; Rejected = canned refusal string.
  • If unfaithful ($h=1$): Chosen = canned refusal string; Rejected = original answer.
  • Direct Preference Optimization (DPO): The model is fine-tuned using DPO on the constructed preference pairs.
[RAG Logs: Query + Reference + Output]
                 │
        (Active Learning / ras)
                 ▼
     [Selected Informative Subset]
                 │
      (Reliability Check h ∈ {0,1})
                 ▼
  [Preference Pairs: Output vs Refusal]
                 │
           (DPO Fine-Tuning)
                 ▼
   [Calibrated, Trustworthy RAG LLM]

Datasets

The authors utilized RAGTruth, a benchmark covering Question Answering, Summarization, and Data-to-Text tasks.

  • Because RAGTruth lacked generations from modern models, the authors generated answers using Llama-3-8B-Instruct and Qwen2.5-7B-Instruct.
  • Outputs were annotated for faithfulness using DeepSeek-v3 (with a 3-run majority vote, followed by manual review for conflicts), yielding ~3,000 preference pairs per model split 8:1:1 into train/val/test sets.
  • Experiments also validated generalization using Llama-2-7B-chat.

Key Insights and Ideas

  • Surface features beat semantic embeddings for AL diversity: Counterintuitively, sparse TF-IDF vectorization outperformed modern dense embedding models (Sentence-BERT and Stella-1.5B). Dense encoders project superficially different phrasing into nearby semantic clusters, whereas TF-IDF better isolates fine-grained lexical variations that often dictate RAG factual boundaries.
  • Model answers cause interference during selection: Ablation studies revealed that computing similarity over query + reference yields peak performance. Factoring the model's generated answer into the distance metric degraded performance, acting as noise.
  • Less data is often better: Models fine-tuned via DPO on 12.5% to 25% of actively selected data outperformed models trained on 100% of the full dataset. Full training datasets introduce noisy, marginal examples that dilute the model's decision boundary for when to refuse.

Key Results

Under identical data budgets, AL4RAG with ras consistently outperformed random sampling, uncertainty-based AL (Entropy, BLEUVar), and standard diversity baselines (Coreset, Naive IDDS, LDCAL):

  • Rejection Performance (Llama-3-8B @ 12.5% data):
  • AL4RAG$_{\text{ras}}$: 31.37% Rejection Rate (RR) / 35.37 F1.
  • Random Selection: 23.04% RR / 28.25 F1.
  • Naive IDDS: 25.00% RR / 30.32 F1.
  • Answer Stability & Faithfulness (Llama-3-8B @ 12.5% data):
  • AL4RAG$_{\text{ras}}$ achieved 75.11% faithfulness (evaluated via Spark X1), outperforming both random selection (74.53%) and prompt-concatenated IDDS (73.87%), alongside higher ROUGE-L (34.15 vs. 33.83).
  • These gains remained consistent across base architectures, including Qwen2.5-7B-Instruct (37.83 F1 at 12.5% vs. 32.73 for Random) and Llama-2-7B-chat.

Limitations and Future Work

  • Privacy in Conversation Logs: Sourcing training data directly from real-world user logs introduces data privacy concerns and potential leakage of sensitive user details if privacy filtering is absent.
  • Synthetic / LLM Annotation Dependency: Budget limitations led the authors to use DeepSeek-v3 as an LLM annotation proxy for the primary labeling pass, which may inherit model-specific evaluation biases despite majority voting.
  • Refusal Granularity: The current setup relies on a binary refusal mechanism with template strings rather than teaching the model to formulate nuanced partial answers or request clarifications.

Summary updated

Abstract

Retrieval-augmented generation (RAG) is a key technique for leveraging external knowledge and enhancing the factual accuracy of large language models (LLMs). However, RAG still faces challenges in ensuring fully reliable responses in all scenarios. To address this, it is essential to identify samples that tend to lead to unreliable outputs or guide LLMs toward factually correct responses, which experts then annotate to develop high-quality datasets for refining LLMs. However, the growing scarcity of such datasets makes their creation challenging. This paper proposes using the vast amount of conversations generated from widespread LLM usage to build these datasets, with the goal of training LLMs to appropriately handle queries outside its capabilities while providing accurate responses to manageable ones. Given the impracticality of having experts annotate all conversation records, we introduce AL4RAG, a framework that uses active learning to select the most suitable conversation samples for annotation, thereby optimizing model performance within a limited annotation budget. Additionally, recognizing that traditional active learning methods are not fully compatible with RAG due to unsuitable distance metrics, we develop a novel sample distance measurement for RAG active learning. Extensive experiments show that our method consistently outperforms baselines across multiple metrics.
BibTeX
@misc{geng2025enhancing,
  author = {Xuzhao Geng and Haozhao Wang and Jun Wang and Wei Liu and Zhang Jinghua and Ruixuan Li},
  title = {{Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capables}},
  year = {2025},
  eprint = {2502.09073},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.09073},
}
Figure