SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales

Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, Jing Gao

TL;DR

This paper introduces SaySelf, a novel framework that trains Large Language Models (LLMs) to not only provide more accurate, fine-grained confidence scores for their answers but also to generate "self-reflective rationales." These rationales explain why the LLM is uncertain, helping to build trust and improve AI reliability.

What Problem Does This Paper Address?

LLMs are incredibly powerful, but they have a significant blind spot: they often struggle to express their confidence in their own answers. This means they can confidently present incorrect or fabricated information (hallucinations) without any indication of doubt. This lack of self-awareness severely limits their trustworthiness and applicability in real-world scenarios where knowing when an LLM doesn't know is as crucial as knowing what it does know. Existing methods for eliciting confidence either have limited performance, provide only coarse-grained (e.g., binary) confidence levels, or significantly increase inference time.

Methods

SaySelf tackles this problem in two main stages:

  1. Supervised Fine-Tuning (SFT) with Self-Reflective Rationales:
    • The core idea here is to create a specialized dataset for fine-tuning LLMs.
  2. For a given question, the LLM generates multiple reasoning chains and answers.
  3. These responses are clustered based on semantic similarity to identify distinct lines of reasoning.
  4. An off-the-shelf LLM (like GPT-4) is then used to analyze these clusters. It's prompted to act as if it were the model being trained and summarize the inconsistencies it finds across different reasoning paths. This summary becomes the "self-reflective rationale," explaining the model's uncertainty from its own perspective.
  5. Confidence scores are derived based on the consistency of responses within clusters.
  6. This process creates a dataset of (question, answer, self-reflective rationale, confidence score) tuples for SFT.
  7. Reinforcement Learning (RL) for Confidence Calibration:
    • While SFT improves confidence estimation, RL is used to further refine and calibrate these scores.
  8. A custom reward function is designed to incentivize accurate, high-confidence predictions while penalizing overconfidence in incorrect answers. The reward is quadratic, meaning larger deviations from perfect calibration (either too confident when wrong, or not confident enough when right) are penalized more heavily.
  9. The Proximal Policy Optimization (PPO) algorithm is used to train the LLM with this reward function.

Datasets

The paper primarily uses the HotpotQA dataset for training and evaluation. This dataset is known for requiring multi-hop reasoning. They also evaluate on several other established datasets to demonstrate the generalizability of SaySelf:

  • TruthfulQA: Tests for truthful answers to questions designed to elicit falsehoods.
  • StrategyQA: Requires multi-hop reasoning for true/false questions.
  • FEVER: Evaluates fact verification against Wikipedia.
  • HaluEval: Specifically designed to evaluate LLM hallucination.
  • ParaRel: Assesses understanding of paraphrased relational facts.
  • SQuADRUN: Used to evaluate performance on unanswerable questions.

No new benchmarks were introduced; instead, they leverage existing ones to showcase SaySelf's effectiveness.

Key Insights and Ideas

  • Rationales as Uncertainty Indicators: The most significant insight is that LLMs can be trained to articulate why they are uncertain. This goes beyond just a number and provides valuable context for users.
  • Self-Reflection for Calibration: By framing the rationale generation as a "self-reflection" process (where the LLM analyzes its own potential outputs), the method encourages a more genuine representation of uncertainty.
  • Bridging SFT and RL: The two-stage approach effectively combines the data-generation power of SFT with the fine-tuning precision of RL for robust confidence calibration.
  • Reduced Inference Latency: Unlike methods that rely on multiple sampling at inference time, SaySelf generates confidence scores and rationales directly after training, leading to lower latency.

Key Results

  • Reduced Calibration Error: SaySelf significantly outperforms various baselines in reducing the Expected Calibration Error (ECE) across multiple datasets. For example, on HotpotQA, SaySelf achieves an ECE of 0.3558, which is statistically significantly better than most baselines.
  • Improved Distinction: The Area Under the Receiver Operating Characteristic Curve (AUROC) also shows improvement, indicating SaySelf's better ability to distinguish between correct and incorrect responses.
  • Maintained Task Performance: Crucially, SaySelf achieves these confidence calibration improvements without sacrificing task accuracy. For instance, on HotpotQA, SaySelf achieves an accuracy of 0.3585, comparable to or better than many baselines.
  • Faithful Rationales: Human evaluations (and GPT-4 evaluations correlated with human ratings) show that the generated self-reflective rationales are faithful and effectively capture internal uncertainty.
  • Low Confidence on Unanswerable Questions: SaySelf demonstrates a significant confidence gap between answerable and unanswerable questions, indicating it can better identify when it doesn't know the answer.

Limitations and Future Work

  • Dependence on Multiple Samples: A potential limitation is SaySelf's reliance on multiple sampled reasoning chains for training. The faithfulness of these chains (especially Chain-of-Thought) is an ongoing research topic.
  • Over-reliance Risk: While SaySelf aims to build trust, there's a risk users might over-rely on the confidence estimates, potentially leading to harmful decisions if the rationales are inaccurate.
  • Future Work: The authors suggest integrating methods to improve the faithfulness of CoT reasoning and further exploring how to enhance the effectiveness and faithfulness of SaySelf. They also highlight the importance of users being aware of the potential for misinterpretations.

Summary updated

Abstract

Large language models (LLMs) often generate inaccurate or fabricated information and generally fail to indicate their confidence, which limits their broader applications. Previous work elicits confidence from LLMs by direct or self-consistency prompting, or constructing specific datasets for supervised finetuning. The prompting-based approaches have inferior performance, and the training-based approaches are limited to binary or inaccurate group-level confidence estimates. In this work, we present the advanced SaySelf, a training framework that teaches LLMs to express more accurate fine-grained confidence estimates. In addition, beyond the confidence scores, SaySelf initiates the process of directing LLMs to produce self-reflective rationales that clearly identify gaps in their parametric knowledge and explain their uncertainty. This is achieved by using an LLM to automatically summarize the uncertainties in specific knowledge via natural language. The summarization is based on the analysis of the inconsistency in multiple sampled reasoning chains, and the resulting data is utilized for supervised fine-tuning. Moreover, we utilize reinforcement learning with a meticulously crafted reward function to calibrate the confidence estimates, motivating LLMs to deliver accurate, high-confidence predictions and to penalize overconfidence in erroneous outputs. Experimental results in both in-distribution and out-of-distribution datasets demonstrate the effectiveness of SaySelf in reducing the confidence calibration error and maintaining the task performance. We show that the generated self-reflective rationales are reasonable and can further contribute to the calibration. The code is made public at https://github.com/xu1868/SaySelf.
BibTeX
@inproceedings{xu2024sayself,
  author = {Tianyang Xu and Shujin Wu and Shizhe Diao and Xiaoze Liu and Xingyao Wang and Yangyi Chen and Jing Gao},
  title = {{SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=LA94kxYCim},
}
Figure