ConfRAG: Confidence-Guided Retrieval-Augmenting Generation

Yin Huang, Yifan Ethan Xu, Kai Sun, Vera Yan, Alicia Yi Sun, Haidar Khan, Jimmy Nguyen, Jingxiang Chen, Mohammad Kachuee, Zhaojiang Lin, Yue Liu, Aaron Colak, Anuj Kumar, Wen-tau Yih, Xin Luna DongSubmission to ICLR 2026 · Withdrawn

TL;DR

Always-on Retrieval-Augmented Generation (RAG) adds unnecessary latency and computational expense, yet Large Language Models (LLMs) are notoriously overconfident and hallucinate when relying on internal memory. ConfRAG resolves this tension by fine-tuning LLMs on simple atomic facts paired with a "dampener" prompt, teaching them to admit "I am unsure" when they lack knowledge. This slashes hallucination rates below 5% and triggers RAG only when needed, reducing median retrieval latency by over 600 ms while matching or beating always-on RAG accuracy.

What Problem Does This Paper Address?

RAG grounds LLM outputs in verified external documents, but calling external search APIs and processing retrieved contexts for every query is computationally expensive and slow. Ideally, a model should answer from its internal parametric weights when confident, deferring to external retrieval only when uncertain.

However, existing RAG triggers are either coarse (manually rule-based), impractical for production (requiring white-box inspection of internal hidden-state entropy at every token), or ineffective (relying on self-reported confidence, where models systematically overestimate what they know). The authors address a core question: Can we teach an LLM to reliably recognize its own factual knowledge boundaries and trigger RAG strictly on demand?

Methods

The proposed framework consists of two core components:

User Query (Static) 
    ├──> Direct LLM (ConfQA) ───[Produces Valid Answer]───> Early-stop RAG & Return Answer
    └──> RAG Pipeline (Parallel) ──[ConfQA says "Unsure"]───> Await & Return RAG Result
  1. ConfQA (Confidence-Calibrated Fine-Tuning):
  2. Targeted Calibration Data: The model is fine-tuned on 3,000 simple, atomic factual question-answer pairs derived from DBpedia (balanced across head, torso, and tail entities).
  3. Selective Abstention: The base model (e.g., Llama-3.1-70B) generates responses to these questions, and an LLM judge (Llama-3.1-405B) grades correctness. If the model answers correctly, the target label is the ground-truth answer; if incorrect, the target label is set to "I am unsure about the answer".
  4. Dampening Prompt: A specific system prompt—"Answer only if you are confident; otherwise, respond with 'I am unsure about the answer'"—is applied during both fine-tuning and inference to reinforce conservative generation.
  5. Atomic Fact Regularization: By focusing fine-tuning purely on single-attribute factual assertions rather than complex multi-step reasoning, the model learns factual confidence boundaries without confusing auxiliary skills.
  6. ConfRAG (Triggering Pipeline):
  7. For static factual questions, ConfQA generation runs in parallel with the initial stages of a RAG pipeline.
  8. If ConfQA emits a factual answer, the external retrieval process is immediately aborted, saving latency and compute.
  9. If ConfQA returns "I am unsure", the system waits for the RAG pipeline to complete and returns the retrieved response.

Datasets

The authors evaluated ConfQA and ConfRAG across several established benchmarks:

  • Short-Form Factuality: Head-to-Tail (1,200 simple attribute queries across DBpedia and IMDb), SimpleQA (4,326 challenging, OpenAI-curated fact-seeking questions), and CRAG (642 static multi-domain questions).
  • Long-Form Factuality: LongFact (250 object queries), AlpacaFact (241 fact-seeking instructions), and Biography (183 Wikipedia person prompts), evaluated using FactScore / VeriScore.
  • General Knowledge & Reasoning: MMLU (5-shot, 14,042 questions) and MMLU-Pro (12,082 multi-choice questions) to verify that refusal fine-tuning does not cause capability collapse.

Key Insights and Ideas

  • LLM Overconfidence is Pervasive: Calibrated probability curves reveal that self-reported LLM confidence is wildly miscalibrated. On CRAG, when Llama-3.1-70B reports an 80% confidence level, its actual accuracy is only 33%. Smaller models are frequently more overconfident than larger ones.
  • Don't Feed New Knowledge During Refusal Tuning: Prior works often train models on ground-truth answers for questions the model got wrong. The authors show this degrades calibration: feeding facts during supervised fine-tuning (SFT) causes massive hallucination (up to 50–70% incorrect rates). Models must be taught to refuse, not forced to swallow unseen facts in SFT.
  • The Synergy of Dampening Prompts and SFT: SFT alone or prompting alone yields inconsistent benefits. Combining atomic-fact refusal training with the dampening system prompt reduces hallucinations by an additional 5–11% over fine-tuning alone.
  • Generalization from Micro-Data: Fine-tuning on just 3,000 DBpedia attribute questions generalized effectively to long-form factuality and complex question answering benchmarks without regressing general reasoning (MMLU score remained stable at ~82.8%).

Key Results

  • Hallucination Elimination: Across short-form benchmarks, ConfQA reduced hallucination rates from 20–40% down to below 5% (e.g., dropping from 26.0% to 5.2% on DBpedia, and 25.7% to 4.4% on CRAG).
  • Optimal Triggering: ConfRAG attained superior triggering $F_1$ scores over state-of-the-art refusal methods (R-Tuning, IDK), achieving 82.4% on DBpedia, 91.3% on SimpleQA, and 81.1% on CRAG.
  • Latency Reduction Without Quality Loss: On the CRAG benchmark, ConfRAG achieved comparable or better accuracy than invoking RAG unconditionally (62.3% vs. 61.1%), while slashing P50 latency by 622 ms (1,278 ms vs. 1,900 ms) and P90 latency by 825 ms.

Limitations and Future Work

  • SFT vs. Preference Optimization: The current framework relies solely on Supervised Fine-Tuning; exploring Direct Preference Optimization (DPO) or Reinforcement Learning from Human/AI Feedback (RLHF) could sharpen boundary recognition further.
  • Open-Weights Requirement: ConfQA requires model parameter updates, meaning it cannot be applied off-the-shelf to black-box commercial APIs (e.g., proprietary Claude or GPT endpoints).
  • Scope Beyond Factual QA: The current calibration focuses strictly on static factual statements; extending confidence calibration to code generation, mathematical proofs, and complex multi-step reasoning remains an open direction.

Summary updated

Abstract

Can Large Language Models (LLMs) be trained to avoid hallucinating factual statements, and can Retrieval-Augmented Generation (RAG) be triggered only when necessary to reduce retrieval and computation costs? In this work, we address both challenges simultaneously. We introduce ConfQA, a fine-tuning strategy that reduces hallucination rates from 20–40% to below 5% across multiple factuality benchmarks. The approach is simple: when the model answers correctly, it is trained to output the answer; otherwise, it is trained to respond with “I am unsure.” Two design choices make this training effective: (1) a dampening prompt (“answer only if you are confident”) that explicitly discourages overconfident hallucinations, and (2) training data drawn from atomic factual statements (e.g., knowledge graph attribute values), which calibrates model confidence and yields robust generalization across domains and question types. Building on ConfQA, we propose ConfRAG, a triggering strategy that invokes RAG only when the model responses with unsure. This framework achieves accuracy above 95% in ideal case while reducing unnecessary external retrievals by over 30%.
BibTeX
@misc{huang2025confrag,
  author = {Yin Huang and Yifan Ethan Xu and Kai Sun and Vera Yan and Alicia Yi Sun and Haidar Khan and Jimmy Nguyen and Jingxiang Chen and Mohammad Kachuee and Zhaojiang Lin and Yue Liu and AARON COLAK and Anuj Kumar and Wen-tau Yih and Xin Luna Dong},
  title = {{ConfRAG: Confidence-Guided Retrieval-Augmenting Generation}},
  year = {2025},
  eprint = {2506.07309},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.07309},
}
Figure