Abstract
Can Large Language Models (LLMs) be trained to avoid hallucinating factual statements, and can Retrieval-Augmented Generation (RAG) be triggered only when necessary to reduce retrieval and computation costs? In this work, we address both challenges simultaneously. We introduce ConfQA, a fine-tuning strategy that reduces hallucination rates from 20-40% to below 5% across multiple factuality benchmarks. The approach is simple: when the model answers correctly, it is trained to output the answer; otherwise, it is trained to respond with "I am unsure". Two design choices make this training effective: (1) a dampening prompt ("answer only if you are confident") that explicitly discourages overconfident hallucinations, and (2) training data drawn from atomic factual statements (e.g., knowledge graph attribute values), which calibrates model confidence and yields robust generalization across domains and question types. Building on ConfQA, we propose ConfRAG, a triggering strategy that invokes RAG only when the model responses with unsure. This framework achieves accuracy above 95% in ideal case while reducing unnecessary external retrievals by over 30%.
What Problem Does This Paper Address?
Large Language Models (LLMs) are notoriously overconfident; they would often rather "hallucinate" a plausible-sounding lie than admit they don't know a fact. To fix this, researchers typically use Retrieval-Augmented Generation (RAG), which anchors model responses in external data (like Wikipedia or internal docs).
However, RAG is a double-edged sword. It is computationally expensive, increases latency, and is often unnecessary if the model already "knows" the answer in its weights. The core challenge is triggering: how do we decide when to invoke RAG? Existing methods are either too complex (monitoring internal "hidden states" of the model) or too blunt (manually turning RAG on for specific domains). The authors seek a lightweight, "fact-level" confidence mechanism that allows the model to self-regulate.
Methods
The researchers propose a two-part solution: ConfQA (the training) and ConfRAG (the architecture).
1. ConfQA: The "I Am Unsure" Fine-Tuning
The authors fine-tuned a Llama-3.1-70B model using a simple but effective strategy. If the model's internal knowledge produces a correct answer, it is trained to output that answer. If it is incorrect or doesn't know, it is trained to respond with a specific string: "I am unsure about the answer."
Two specific design choices made this work:
- The Dampening Prompt: They used a system instruction—"Answer only if you are confident"—during both training and inference. This explicitly discourages the model from guessing.
- Atomic Fact Training: Instead of training on complex reasoning, they used "atomic" facts (simple entity-attribute pairs from DBPedia, like a movie's director). This focused the model on factual calibration rather than linguistic style.
2. ConfRAG: Parallel Triggering
In the ConfRAG framework, the system runs the standard LLM generation and the RAG pipeline in parallel. If the LLM generates a confident answer, the RAG process is "early-stopped" to save time and compute. If the LLM says "I am unsure," the system waits for and returns the RAG result.
Datasets
The authors evaluated their approach across seven diverse benchmarks to test generalization:
- Short-form Factuality: Head-to-Tail (DBPedia/IMDb), SimpleQA (OpenAI’s factuality set), and CRAG (Comprehensive RAG benchmark).
- Long-form Factuality: LongFact, AlpacaFact, and Biography.
- General Knowledge: MMLU and MMLU-Pro.
For training, they used a small, high-quality set of 3,000 samples derived from DBPedia, specifically balanced across "head" (popular), "torso," and "tail" (obscure) entities.
Key Insights and Ideas
- Self-Reported Confidence is a Lie: The paper confirms that when you ask an LLM "How confident are you on a scale of 0 to 1?", it almost always overestimates its accuracy. You cannot trust a model's raw probability scores for RAG triggering.
- Smaller Models are Cockier: Interestingly, the authors found that within the same model family, smaller models are often more overconfident than larger ones—a phenomenon they link to a "correlation between ignorance and self-assurance."
- Atomic Facts Generalize: One of the biggest "aha" moments was that training a model to be humble on simple DBPedia facts successfully taught it to be humble on complex, multi-hop reasoning tasks in other datasets.
- The Power of the Dampener: The "dampening prompt" acts as a crucial regularization signal. Without it, the model's hallucination rate remains significantly higher.
Key Results
- Hallucination Collapse: ConfQA reduced hallucination rates from a baseline of 20–40% down to below 5% across multiple benchmarks.
- Efficiency Gains: ConfRAG reduced unnecessary external retrievals by 30–50% compared to a "RAG-always" approach.
- Latency Improvement: On the CRAG benchmark, the system reduced P50 latency by over 600ms, as the model could skip the retrieval step for facts it already knew.
Limitations and Future Work
- SFT vs. DPO: The current work relies on Supervised Fine-Tuning (SFT). The authors suggest that Direct Preference Optimization (DPO) might further refine the boundary between "sure" and "unsure."
- Scope of Knowledge: The study focused primarily on static, factual knowledge. Extending this to dynamic facts (real-time news) or specialized domains like math and coding is a necessary next step.
- Model Access: This method requires the ability to fine-tune the model. For developers using closed-source APIs where fine-tuning isn't an option, the specific "ConfQA" training cannot be applied directly.
Summary updated