Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism

Lang Cao

Abstract

Large language models (LLMs) possess remarkable language understanding and generation capabilities, yet they are prone to "hallucinations"—generating factually incorrect or misleading information. This paper addresses this reliability gap by introducing the Learn to Refuse (L2R) framework. L2R mitigates hallucinations in question-answering tasks by implementing a dual-stage refusal mechanism and a decoupled, structured knowledge base. By treating the LLM as a reasoning engine rather than a database, the system can recognize its knowledge boundaries and abstain from answering when evidence is insufficient. Our results demonstrate that L2R significantly enhances the controllability and accuracy of LLMs across general and specialized domains.

Introduction: The Challenge of Hallucination

Despite their utility, standard LLMs are designed to be helpful, often leading them to attempt answers even when they lack the necessary information. These fact-conflicting hallucinations occur when a model's internal weights contain incomplete or outdated knowledge. While Retrieval-Augmented Generation (RAG) provides external context, models still struggle to identify the limits of that context. The core objective of this research is to provide LLMs with a formal mechanism to say "I don't know," thereby increasing the trustworthiness of their outputs.

The Learn to Refuse (L2R) Framework

The L2R framework shifts the paradigm from relying on an LLM's internal knowledge to utilizing a traceable, external source. The system is built upon three primary pillars:

  1. Knowledge Scope Limitation: The LLM is treated as a "zero-knowledge" processor. It is restricted to answering questions solely based on an independent, structured Knowledge Base (KB) that can be progressively expanded and validated.
  2. Automatic Knowledge Enrichment (AKE): To populate the KB efficiently, L2R employs a multi-agent pipeline. A Question Generation Agent creates seed queries, an Answer Generation Agent provides responses with confidence scores, and a QA-to-Knowledge Agent distills high-confidence pairs into clean, declarative facts.
  3. Dual Refusal Mechanism: Reliability is enforced through two layers of filtering. Soft Refusal involves the LLM judging its own ability to answer based on retrieved context. Hard Refusal acts as a system-level safety net, calculating semantic similarity between the query and retrieved knowledge; if the score falls below a defined threshold, a refusal is triggered automatically.
The architecture of the Learn-to-Refuse (L2R) system, highlighting the knowledge enrichment pipeline and the dual-stage refusal process during question answering.

Experimental Evaluation and Results

We evaluated the L2R framework across several rigorous benchmarks, including TruthfulQA (measuring mimicry of human falsehoods), CommonsenseQA, and MedQA (specialized medical knowledge).

Performance comparison of L2R against baseline models across multiple benchmarks, showing significant accuracy gains in TruthfulQA and MedQA.

As shown in the results, L2R-GPT achieved a substantial accuracy of 65.1% on TruthfulQA-MC1, a significant improvement over the base GPT-3.5 (46.6%) and standard RAG (53.7%). This performance boost is directly attributed to the model's ability to refuse approximately 20% of the questions where it lacked sufficient evidence. Similar improvements were observed in the medical domain (MedQA), where accuracy rose from 51.2% to 52.8% upon integrating a structured medical corpus.

Key Insights and Discussion

  • Decoupling Reasoning from Knowledge: By isolating the reasoning engine from the data source, L2R provides full traceability. Every answer can be linked back to a specific entry in the KB, making the system's logic transparent.
  • The Necessity of Hard Refusal: Ablation studies confirmed that LLMs often overestimate their own knowledge. The mathematical "hard refusal" threshold is essential to prevent the model from attempting to answer when retrieved data is irrelevant.
  • Accuracy-Coverage Trade-off: L2R highlights a fundamental trade-off in AI reliability: by refusing to answer difficult or uncertain questions, the total count of answered questions decreases, but the precision of the remaining answers increases dramatically.

Conclusion and Future Work

The Learn to Refuse framework represents a significant step toward more controllable and reliable AI. While currently optimized for "one-hop" factual questions, future work will focus on scaling the system to million-scale knowledge bases and adapting the refusal philosophy to more complex reasoning tasks and creative writing domains where "correctness" is less binary. By teaching models when to remain silent, we can ensure that when they do speak, they are correct.

Summary updated

Abstract

Large language models (LLMs) have demonstrated impressive language understanding and generation capabilities, enabling them to answer a wide range of questions across various domains. However, these models are not flawless and often produce responses that contain errors or misinformation. These inaccuracies, commonly referred to as hallucinations, render LLMs unreliable and even unusable in many scenarios. In this paper, our focus is on mitigating the issue of hallucination in LLMs, particularly in the context of question-answering. Instead of attempting to answer all questions, we explore a refusal mechanism that instructs LLMs to refuse to answer challenging questions in order to avoid errors. We then propose a simple yet effective solution called Learn to Refuse (L2R), which incorporates the refusal mechanism to enable LLMs to recognize and refuse to answer questions that they find difficult to address. To achieve this, we utilize a structured knowledge base to represent all the LLM’s understanding of the world, enabling it to provide traceable gold knowledge. This knowledge base is separate from the LLM and initially empty. It can be filled with validated knowledge and progressively expanded. When an LLM encounters questions outside its domain, the system recognizes its knowledge scope and determines whether it can answer the question independently. Additionally, we introduce a method for automatically and efficiently expanding the knowledge base of LLMs. Through qualitative and quantitative analysis, we demonstrate that our approach enhances the controllability and reliability of LLMs.
BibTeX
@inproceedings{cao2024learn,
  author = {Lang Cao},
  title = {{Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=fqu5C03rfR},
}
Figure