SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales

Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, Jing Gao

Abstract

Large language models (LLMs) often generate inaccurate or fabricated information and generally fail to indicate their confidence, which limits their broader applications. Previous work has elicited confidence from LLMs by direct or self-consistency prompting, or constructing specific datasets for supervised finetuning. The prompting-based approaches have inferior performance, and the training-based approaches are limited to binary or inaccurate group-level confidence estimates. In this work, we present SaySelf, a novel training framework that teaches LLMs to express more fine-grained confidence estimates. In addition, beyond the confidence scores, SaySelf initiates the process of directing LLMs to produce self-reflective rationales that clearly identify gaps in their parametric knowledge and explain their uncertainty. This is achieved by using an LLM to automatically summarize the uncertainties in specific knowledge via natural language. The summarization is based on the analysis of the inconsistency in multiple sampled reasoning chains, and the resulting data is utilized for supervised fine-tuning. Moreover, we utilize reinforcement learning with a meticulously crafted reward function to calibrate the confidence estimates, motivating LLMs to deliver accurate, high-confidence predictions and to penalize overconfidence in erroneous outputs. Experimental results demonstrate the effectiveness of SaySelf in reducing the confidence calibration error and maintaining the task performance. The generated self-reflective rationales are also reasonable and can further contribute to the calibration.

Introduction

Large Language Models (LLMs) are notorious for "hallucinating"—generating facts that sound plausible but are entirely fabricated. A major part of this problem is calibration: LLMs often do not know when they are guessing. When asked a question outside their training data, they tend to provide an answer with the same authoritative tone as a well-known fact. Existing solutions generally fall into two camps: prompting-based methods, which are often poorly calibrated or prohibitively slow, and training-based methods, which often result in coarse binary outputs that lack nuance.

SaySelf is a novel training framework that enables LLMs to produce fine-grained confidence estimates accompanied by "self-reflective rationales"—natural language explanations of their own uncertainty. By combining model-specific supervised fine-tuning with a specialized reinforcement learning reward function, SaySelf significantly reduces calibration error and helps models identify their own knowledge gaps without increasing inference-time latency.

Comparison of SaySelf with previous confidence elicitation methods like Direct Prompting, Self-Consistency, and R-Tuning.

Methodology

The SaySelf framework consists of two primary stages designed to bridge the gap between model output and internal certainty.

Supervised Fine-Tuning (SFT) with Self-Reflective Rationales

To teach a model about its own blind spots, the authors first create a model-specific dataset. They generate 100 different reasoning chains for a single question and cluster these responses based on semantic similarity. If the model is certain, the responses are consistent; if confused, they split into different clusters. A teacher model (GPT-4) analyzes these clusters and writes a self-reflective rationale summarizing the inconsistency (e.g., "I am uncertain because I have conflicting information about the current President"). The model is then fine-tuned on this data to predict the answer, the rationale, and a confidence score based on the cluster distribution.

Reinforcement Learning (RL) from Task Supervision

To sharpen calibration, the authors use Reinforcement Learning (specifically PPO) with a quadratic reward function. This function is designed to penalize the model heavily for being "confident but wrong" and reward it for being "confident and right." This forces the model to differentiate its confidence levels more accurately across different types of questions, moving beyond the homogeneous confidence levels often produced by SFT alone.

Evaluation and Key Results

The authors evaluated SaySelf using several knowledge-intensive benchmarks, including HotpotQA, TruthfulQA, HaluEval, and StrategyQA.

  • Reduced Calibration Error: SaySelf significantly outperformed baselines in Expected Calibration Error (ECE). For example, on the HaluEval dataset, SaySelf achieved an ECE of 0.2661, compared to 0.6746 for direct prompting.
  • Better Distinction: Using the AUROC metric, SaySelf showed a marked improvement in distinguishing between its own correct and incorrect answers, often exceeding baselines by 10-20%.
  • Efficiency at Inference: Unlike self-consistency methods that require sampling dozens of responses to estimate confidence, a SaySelf-trained model generates its answer, rationale, and confidence in a single forward pass.
  • Handling the Unknown: On the SQUADRUN "unanswerable" task, SaySelf demonstrated a much larger "confidence gap" between answerable and unanswerable questions, effectively flagging when a question is impossible to answer based on its training.

Conclusion

SaySelf demonstrates that forcing a model to explain its uncertainty through self-reflective rationales improves the accuracy of the confidence score itself. By building data based on the specific inconsistencies of the model being trained, SaySelf acknowledges that different models have different "knowledge holes." While the training process is computationally expensive, the resulting model is highly efficient at inference time, providing a scalable path toward more reliable and self-aware AI systems.

Summary updated

Abstract

Large language models (LLMs) often generate inaccurate or fabricated information and generally fail to indicate their confidence, which limits their broader applications. Previous work has elicited confidence from LLMs by direct or self-consistency prompting, or constructing specific datasets for supervised finetuning. The prompting-based approaches have inferior performance, and the training-based approaches are limited to binary or inaccurate group-level confidence estimates. In this work, we present SaySelf, a novel training framework that teaches LLMs to express more fine-grained confidence estimates. In addition, beyond the confidence scores, SaySelf initiates the process of directing LLMs to produce self-reflective rationales that clearly identify gaps in their parametric knowledge and explain their uncertainty. This is achieved by using an LLM to automatically summarize the uncertainties in specific knowledge via natural language. The summarization is based on the analysis of the inconsistency in multiple sampled reasoning chains, and the resulting data is utilized for supervised fine-tuning. Moreover, we utilize reinforcement learning with a meticulously crafted reward function to calibrate the confidence estimates, motivating LLMs to deliver accurate, high-confidence predictions and to penalize overconfidence in erroneous outputs. Experimental results demonstrate the effectiveness of SaySelf in reducing the confidence calibration error and maintaining the task performance. The generated self-reflective rationales are also reasonable and can further contribute to the calibration. The code is made public at https://github.com/xu1868/SaySelf.
BibTeX
@inproceedings{xu2024sayself,
  author = {Tianyang Xu and Shujin Wu and Shizhe Diao and Xiaoze Liu and Xingyao Wang and Yangyi Chen and Jing Gao},
  title = {{SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=LA94kxYCim},
}
Figure