Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting

Guande He, Peng Cui, Jianfei Chen, Wenbo Hu, Jun Zhu

Abstract

Aligned language models (LMs), while more helpful and conversational, often exhibit overconfidence in their outputs compared to their pre-trained counterparts. This study systematically evaluates the impact of alignment on logit-based uncertainty calibration in multiple-choice settings. We identify two distinct uncertainties—answer uncertainty and format uncertainty—and demonstrate that their conflation during alignment leads to poor calibration. We propose a sample-efficient post-hoc calibration method leveraging pre-trained model distributions to restore reliability.

Introduction: The Calibration Gap

Calibration measures how well a model's predicted probabilities align with actual outcomes. In safety-critical domains, a model must "know what it knows." While pre-trained LMs are often well-calibrated when prompted with few-shot examples, the alignment process—Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF)—typically degrades this property. Aligned models frequently assign near-100% confidence to incorrect answers.

Reliability Diagram and Confidence Histogram comparing Llama-1 and Vicuna-v1.3. Aligned models like Vicuna show a significant shift toward high confidence and a deviation from perfect calibration.

Uncertainty Decomposition: Answer vs. Format

The core of the calibration issue lies in the decomposition of uncertainty:

  1. Answer Uncertainty: The inherent difficulty of the task or question.
  2. Format Uncertainty: Confusion regarding the expected response structure (e.g., "The answer is A" vs. "(A)").

In-Context Learning (ICL) helps pre-trained models primarily by reducing format uncertainty. In contrast, alignment processes bake a strong format preference into the model's weights. This "format-forcing" leaks into the answer confidence, causing the model to appear certain about the choice itself simply because it is certain about the format.

Systematic Evaluation of Alignment

Our experiments across various model families (Llama, Vicuna, Mistral) reveal a consistent trend: alignment increases Expected Calibration Error (ECE) even when accuracy remains stable or improves.

ECE vs. Accuracy across different model sizes and alignment stages. Aligned models (dashed/solid lines with markers) generally exhibit higher ECE than their pre-trained counterparts at similar accuracy levels.

Synthetic experiments further confirm this "Conflation Trap." When models are trained specifically to change their format (SFT-Format), calibration remains stable. However, training them to be certain about the choice (SFT-Choice) destroys calibration.

Proposed Solution: Few-Shot Post-Hoc Calibration

To address overconfidence, we introduce a calibration method that uses the pre-trained model as an "anchor." Instead of standard temperature scaling, we learn a task-specific temperature parameter by minimizing the KL Divergence between the aligned model's output and the better-calibrated pre-trained model's distribution. This method is highly sample-efficient, requiring as few as 5 examples to significantly reduce ECE on benchmarks like MMLU.

Conclusion

The overconfidence of aligned LMs is a byproduct of conflating format preference with answer certainty. By recognizing the superior calibration of pre-trained ancestors, we can develop post-hoc methods to "re-calibrate" helpful models without sacrificing their conversational utility. Future work should extend these insights to open-ended generation where logit extraction is more complex.

Summary updated

Abstract

Despite the significant progress made in practical applications of aligned language models (LMs), they tend to be overconfident in output answers compared to the corresponding pre-trained LMs. In this work, we systematically evaluate the impact of the alignment process on logit-based uncertainty calibration of LMs under the multiple-choice setting. We first conduct a thoughtful empirical study on how aligned LMs differ in calibration from their pre-trained counterparts. Experimental results reveal that there are two distinct uncertainties in LMs under the multiple-choice setting, which are responsible for the answer decision and the format preference of the LMs, respectively. Then, we investigate the role of these two uncertainties on aligned LM's calibration through fine-tuning in simple synthetic alignment schemes and conclude that one reason for aligned LMs' overconfidence is the conflation of these two types of uncertainty. Furthermore, we examine the utility of common post-hoc calibration methods for aligned LMs and propose an easy-to-implement and sample-efficient method to calibrate aligned LMs. We hope our findings could provide insights into the design of more reliable alignment processes for LMs.
BibTeX
@misc{he2023investigating,
  author = {Guande He and Peng Cui and Jianfei Chen and Wenbo Hu and Jun Zhu},
  title = {{Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting}},
  year = {2023},
  eprint = {2310.11732},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.11732},
}
Figure