Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting

Guande He, Peng Cui, Jianfei Chen, Wenbo Hu, Jun ZhuSubmission to ICLR 2024 · Rejected

TL;DR

While instruction tuning and RLHF make language models much more helpful, they notoriously ruin their uncertainty calibration, making aligned models dangerously overconfident. This paper reveals that alignment conflates format uncertainty (how to structure a response) with answer uncertainty (which candidate is correct), artificially inflating confidence on answer tokens. To fix this without massive calibration sets, the authors propose an easy, 5-shot post-hoc temperature scaling method that re-aligns the model’s predictions with its well-calibrated pre-trained counterpart.

What Problem Does This Paper Address?

For language models (LMs) deployed in safety-critical domains (such as medicine or law), knowing what the model does not know is vital. A model is well-calibrated if its predictive confidence matches its empirical accuracy (e.g., predictions made with 80% confidence should be correct 80% of the time).

Previous work observed a frustrating paradox: large pre-trained base LMs are surprisingly well-calibrated out of the box, but aligned LMs (fine-tuned via Supervised Fine-Tuning [SFT] or Reinforcement Learning from Human Feedback [RLHF/DPO]) become severely miscalibrated and overconfident. This paper systematically investigates why alignment breaks logit-based calibration under the multiple-choice question (MCQ) setting and explores how to restore it efficiently.

Methods

1. Decomposing Predictive Uncertainty

The authors formalize an LM’s predictive probability $p_\theta(y \mid x)$ into two components: $$p_\theta(y \mid x) = \underbrace{p_\theta(y \mid x, F)}_{\text{Answer Uncertainty}} \cdot \underbrace{p_\theta(F \mid x)}_{\text{Format Uncertainty}}$$ - Format Uncertainty ($F$): The uncertainty regarding how to structure the output (e.g., answering directly with (A) versus rambling or saying "Let's think step by step"). - Answer Uncertainty: The uncertainty over which option is actually correct, conditioned on adopting the multiple-choice format.

2. Controlled Synthetic Alignment

To pinpoint where overconfidence originates, the authors design synthetic alignment experiments on LLaMA-1 (7B) using LoRA. They isolate training objectives across three variants for SFT and DPO:

  • Format-only: Optimizing only the format identifier tokens (e.g., ().
  • Choice-only: Optimizing only the answer choice tokens (e.g., A).
  • Mixed: Optimizing both format and choice simultaneously (standard alignment behavior).

3. Distribution-Matching Post-Hoc Calibration

Traditional post-hoc calibration methods (like standard Temperature Scaling [TS] or Kernel Density Estimation [KDE]) perform poorly when only a handful of calibration examples are available, because sample accuracy on 5 examples is extremely noisy.

The authors propose a sample-efficient Temperature Scaling method: instead of fitting $T$ to ground-truth accuracy labels on a tiny validation set, they minimize the Kullback–Leibler (KL) divergence between the temperature-scaled aligned LM and the calibrated pre-trained LM: $$\min_T \sum_{i=1}^M D_{\text{KL}}\left(p_\theta^{\text{PT}}(\tilde{y} \mid \tilde{x}_i) \parallel p_{\theta, T}(\tilde{y} \mid \tilde{x}_i)\right)$$

Datasets

The authors evaluated models across seven diverse multiple-choice benchmarks:

  • General & Specialized Knowledge: MMLU
  • Commonsense Reasoning: HellaSWAG, OpenBookQA, TruthfulQA
  • Logical Reasoning: LogiQA
  • Safety & Classification: CivilComments (toxicity), IMDB (sentiment)
  • Synthetic Benchmark: A 4-choice word-matching dataset adapted from Lieberum et al. (2023) for controlled alignment experiments.

Evaluated model families included LLaMA-1 (7B–33B), LLaMA-2 (7B–70B), Vicuna-v1.3, LLaMA-2-Chat, Mistral-7B, Zephyr-7B, and Alpaca-Farm.

Key Insights and Ideas

  • Why pre-trained models look well-calibrated in few-shot settings: In zero-shot settings, pre-trained models appear underconfident because their format uncertainty is high (they don't instinctively start answering with choice letters). In-context learning (ICL) doesn't teach the model the underlying knowledge—it simply eliminates format uncertainty, revealing the model's inherently well-calibrated answer distribution.
  • Alignment conflates both uncertainties: Standard alignment routines train models to adopt concise, human-preferred formats while simultaneously selecting the right answer. In doing so, the optimization shifts answer logit distributions upwards indiscriminately across unrelated domains, resulting in widespread overconfidence.
  • Altering format alone preserves calibration: In synthetic experiments, optimizing purely for format preference (DPO-Format) successfully taught the model to output the desired format without degrading calibration on MMLU. Updating choice tokens (Choice or Mixed), however, triggered heavy overconfidence.

Key Results

  • Overconfidence Gap: On MMLU (5-shot), pre-trained LLaMA-1 33B achieved an average confidence of 58.04% (matching its 57.76% accuracy), whereas aligned Vicuna-v1.3 33B scored a similar 58.63% accuracy but an average confidence of 76.30%, spiking the Expected Calibration Error (ECE).
  • ICL Cannot Fix Aligned Overconfidence: While 5-shot ICL brings LLaMA-2 70B's ECE on MMLU down from ~12.1% to 1.91%, LLaMA-2-Chat 70B remains heavily overconfident across both zero-shot and few-shot evaluations.
  • Few-Shot Calibration: Using only 5 validation examples, the authors' proposed KL-divergence temperature scaling consistently outperformed out-of-the-box models, a constant temperature baseline ($T = 2.5$), and standard few-shot TS/KDE across all 7 benchmark tasks on LLaMA-2-Chat 70B.
MMLU Calibration Error (ECE10) Comparison on LLaMA-2-Chat 70B:
• Out-of-the-box:        ~17.0%
• Few-shot TS:           ~14.5%
• Few-shot KDE:          ~22.0%
• Constant TS (T=2.5):   ~11.0%
• Proposed Method (5-shot): ~7.5% (Substantial reduction across all sub-tasks)

Limitations and Future Work

  • Multiple-Choice Constraint: The uncertainty decomposition relies on logit evaluation at discrete token choice positions; extending this framework to free-form, open-ended generation remains challenging.
  • Dependency on Base Models: The proposed post-hoc calibrator requires white-box logit access to the underlying pre-trained base model and assumes that the base model itself is well-calibrated.
  • Scaling to General Alignment: While synthetic tests demonstrated that separating format from answer updates prevents miscalibration, designing realistic RLHF/DPO loss functions that isolate these factors on open-ended conversational datasets remains an unsolved problem.

Summary updated

Abstract

Despite the significant progress made in practical applications of aligned language models (LMs), they tend to be overconfident in output answers compared to the corresponding pre-trained LMs. In this work, we systematically evaluate the impact of the alignment process on logit-based uncertainty calibration of LMs under the multiple-choice setting. We first conduct a thoughtful empirical study on how aligned LMs differ in calibration from their pre-trained counterparts. Experimental results reveal that there are two distinct uncertainties in LMs under the multiple-choice setting, which are responsible for the answer decision and the format preference of the LMs, respectively. Then, we investigate the role of these two uncertainties on aligned LM's calibration through fine-tuning in simple synthetic alignment schemes and conclude that one reason for aligned LMs' overconfidence is the conflation of these two types of uncertainty. Furthermore, we examine the utility of common post-hoc calibration methods for aligned LMs and propose an easy-to-implement and sample-efficient method to calibrate aligned LMs. We hope our findings could provide insights into the design of more reliable alignment processes for LMs.
BibTeX
@misc{he2023investigating,
  author = {Guande He and Peng Cui and Jianfei Chen and Wenbo Hu and Jun Zhu},
  title = {{Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting}},
  year = {2023},
  eprint = {2310.11732},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.11732},
}

Loading related content...

Figure