TL;DR
This paper tackles the critical issue of factual hallucinations in Large Language Models (LLMs) by proposing a novel training framework called Reinforcement Learning from Knowledge Feedback (RLKF). RLKF leverages LLMs' inherent self-awareness of their internal knowledge states to improve their factuality and honesty, moving beyond simply generating plausible-sounding text.
What Problem Does This Paper Address?
Large Language Models (LLMs) like GPT-3 and LLaMA have demonstrated remarkable capabilities, but they suffer from a significant problem: "hallucination," where they generate content that is factually incorrect or misaligned with reality. This is particularly problematic in high-stakes applications like healthcare or law. While existing methods like retrieval augmentation (RAG) can help by providing external knowledge, this paper focuses on a more fundamental approach: improving how LLMs utilize their internal knowledge. The core challenge is that LLMs often possess factual knowledge but fail to express it accurately during generation, leading to hallucinations.
Methods
The paper introduces two main components:
- DreamCatcher: An automated tool for annotating hallucinations. It combines knowledge probing (checking if the model knows something) with consistency checking (seeing if its generated answers are consistent) to rank factual preference data. This tool helps create datasets that highlight factual accuracy over mere plausibility.
- RLKF (Reinforcement Learning from Knowledge Feedback): This is the core training framework. It uses the factual preference data generated by DreamCatcher to train a reward model. This reward model then guides the LLM's learning process through reinforcement learning (specifically, Proximal Policy Optimization - PPO). The goal is to train LLMs to either accurately express their internal knowledge or honestly admit when they don't know, rather than hallucinating.
Datasets
- Wiki-QA: A dataset created by sampling passages from Wikipedia and using GPT-3.5 to generate question-answer pairs. This dataset is used for both training and evaluating the models.
- DreamCatcher Labeled Data: The paper mentions sampling 200 entries (half Chinese, half English) from DreamCatcher's output for human annotation to validate its performance.
Key Insights and Ideas
- LLMs do have self-awareness of their knowledge: Experiments show LLMs can achieve over 85% accuracy in discerning whether they possess internal knowledge about a given fact, even with limited data. This is a crucial finding, as it confirms the potential for leveraging this internal state.
- Self-awareness doesn't automatically mean honesty: Despite knowing facts, LLMs often fail to express this knowledge faithfully during generation, leading to hallucinations. This highlights the gap between internal knowledge and external output.
- Internal representations hold the key: Probing experiments reveal that "hidden state" probes are most effective at discerning the model's knowledge state, suggesting that this information is encoded within the model's internal layers and can be utilized.
- RLKF bridges the gap: The RLKF framework effectively trains LLMs to better utilize their internal knowledge, leading to improved factuality and honesty without necessarily relying on external retrieval.
Key Results
- High knowledge state probing accuracy: LLMs demonstrated over 85% accuracy in identifying their internal knowledge state across various models and internal representations.
- DreamCatcher's effectiveness: The automated hallucination annotation tool achieved 81% agreement with human annotators.
- RLKF improves factuality and honesty: Experiments showed that RLKF-trained models improved performance on various knowledge-based and reasoning tasks. For instance, on TruthfulQA, most models showed an increase in honesty (e.g., Qwen-chat-14B improved from 43.7% to 49.1%). RLKF also avoided the typical "alignment tax" seen with other fine-tuning methods, and even led to improvements on some benchmarks.
Limitations and Future Work
- Data limitations: The RLKF training relied on a relatively limited quantity and variety of data, primarily generated by GPT-3.5 from Wikipedia passages. While this ensures factual precision, it might lack diversity in writing styles. Future work could involve compiling more intricate factual question-answer data.
- Optimization techniques: The paper used PPO for optimization, but other techniques like DPO or reject sampling could be explored. Future research could creatively incorporate factual preference data into various training frameworks.
- Room for improvement: While RLKF shows promise in increasing honesty, some responses to "unknown" questions still indicate uncertainty, and a significant percentage of responses remain hallucinations, suggesting there's still room for improvement.
Summary updated