Abstract
Large language models (LLMs) have demonstrated impressive language understanding and generation capabilities, enabling them to answer a wide range of questions across various domains. However, these models are not flawless and often produce responses that contain errors or misinformation. These inaccuracies, commonly referred to as hallucinations, render LLMs unreliable and even unusable in many scenarios. In this paper, our focus is on mitigating the issue of hallucination in LLMs, particularly in the context of question-answering. Instead of attempting to answer all questions, we explore a refusal mechanism that instructs LLMs to refuse to answer challenging questions in order to avoid errors. We then propose a simple yet effective solution called Learn to Refuse (L2R), which incorporates the refusal mechanism to enable LLMs to recognize and refuse to answer questions that they find difficult to address. To achieve this, we utilize a structured knowledge base to represent all the LLM's understanding of the world, enabling it to provide traceable gold knowledge. This knowledge base is separate from the LLM and initially empty. It can be filled with validated knowledge and progressively expanded. When an LLM encounters questions outside its domain, the system recognizes its knowledge scope and determines whether it can answer the question independently. Additionally, we introduce a method for automatically and efficiently expanding the knowledge base of LLMs. Through qualitative and quantitative analysis, we demonstrate that our approach enhances the controllability and reliability of LLMs.
Introduction
Large Language Models (LLMs) are notorious for "hallucinating"—confidently stating facts that are incorrect or nonsensical. This usually happens because LLMs are trained to be helpful and often attempt to answer every question, even when they lack the specific knowledge required. Current solutions like Retrieval-Augmented Generation (RAG) help, but they don't solve the problem of the model "over-claiming" or ignoring the retrieved context in favor of its own internal weights.
The Learn to Refuse (L2R) framework seeks to transform the LLM from an "all-knowing" oracle into a controllable processing engine that only answers when it has verified evidence. By restricting the model's knowledge scope and implementing a robust refusal mechanism, L2R significantly improves the reliability and traceability of question-answering systems.
Methodology
The L2R framework consists of three main pillars designed to enforce factuality and controllability:
1. Knowledge Scope Limitation
Instead of relying on the LLM's internal parametric knowledge, the system uses an independent, structured Knowledge Base (KB). The LLM is explicitly instructed to act as a "zero-knowledge" agent that only uses information retrieved from this KB. This "blank slate" approach makes the system much more predictable and easier to debug.
2. Dual Refusal Mechanism
L2R implements a two-layered "insurance policy" to prevent hallucinations:
- Soft Refusal: The LLM is prompted to independently judge if the retrieved information is sufficient to answer the question.
- Hard Refusal: A deterministic system-level check that uses a mathematical function to compare the similarity score of retrieved knowledge against a confidence threshold. If the evidence isn't "strong" enough, the system forces a refusal, bypassing the LLM's judgment.
3. Automatic Knowledge Enrichment (AKE)
To avoid the labor-intensive process of manual KB creation, AKE uses specialized LLM agents to generate questions, provide answers with confidence scores, and convert them into structured narrative facts to populate the KB.
4. Traceable Output
When the model does answer, it must provide the specific evidence and the reasoning path used, following a Chain-of-Thought approach. This ensures that every claim made by the system can be audited.
Experimental Results
The framework was evaluated using several prominent benchmarks, including TruthfulQA (MC1 and MC2), CommonsenseQA, and MedQA.
As shown in the table above, L2R demonstrated significant improvements over standard LLMs and basic RAG setups. On the TruthfulQA-MC1 task, L2R-GPT achieved 65.1% accuracy, a substantial jump from the 46.6% achieved by the base GPT-3.5 model and the 53.7% achieved by standard RAG. In the specialized medical domain (MedQA), L2R improved accuracy by 15.9% compared to the Llama-2 baseline when provided with a reliable medical corpus.
Ablation studies confirmed that the refusal mechanisms are critical; removing them led to a noticeable decline in performance, proving that the ability to say "no" is just as important as the ability to answer correctly.
Conclusion and Future Work
The L2R framework addresses the critical issue of hallucination by limiting knowledge scope and enforcing a dual refusal mechanism. While highly effective for Question Answering, future work will address scaling challenges for million-scale databases and improving performance on complex, multi-hop reasoning tasks where synthesizing multiple disparate facts is required.
Summary updated