Abstract
Aligned language models (LMs), while more helpful and conversational, often exhibit overconfidence in their outputs compared to their pre-trained counterparts. This study systematically evaluates the impact of alignment on logit-based uncertainty calibration in multiple-choice settings. We identify two distinct uncertainties—answer uncertainty and format uncertainty—and demonstrate that their conflation during alignment leads to poor calibration. We propose a sample-efficient post-hoc calibration method leveraging pre-trained model distributions to restore reliability.
Introduction: The Calibration Gap
Calibration measures how well a model's predicted probabilities align with actual outcomes. In safety-critical domains, a model must "know what it knows." While pre-trained LMs are often well-calibrated when prompted with few-shot examples, the alignment process—Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF)—typically degrades this property. Aligned models frequently assign near-100% confidence to incorrect answers.
Uncertainty Decomposition: Answer vs. Format
The core of the calibration issue lies in the decomposition of uncertainty:
- Answer Uncertainty: The inherent difficulty of the task or question.
- Format Uncertainty: Confusion regarding the expected response structure (e.g., "The answer is A" vs. "(A)").
In-Context Learning (ICL) helps pre-trained models primarily by reducing format uncertainty. In contrast, alignment processes bake a strong format preference into the model's weights. This "format-forcing" leaks into the answer confidence, causing the model to appear certain about the choice itself simply because it is certain about the format.
Systematic Evaluation of Alignment
Our experiments across various model families (Llama, Vicuna, Mistral) reveal a consistent trend: alignment increases Expected Calibration Error (ECE) even when accuracy remains stable or improves.
Synthetic experiments further confirm this "Conflation Trap." When models are trained specifically to change their format (SFT-Format), calibration remains stable. However, training them to be certain about the choice (SFT-Choice) destroys calibration.
Proposed Solution: Few-Shot Post-Hoc Calibration
To address overconfidence, we introduce a calibration method that uses the pre-trained model as an "anchor." Instead of standard temperature scaling, we learn a task-specific temperature parameter by minimizing the KL Divergence between the aligned model's output and the better-calibrated pre-trained model's distribution. This method is highly sample-efficient, requiring as few as 5 examples to significantly reduce ECE on benchmarks like MMLU.
Conclusion
The overconfidence of aligned LMs is a byproduct of conflating format preference with answer certainty. By recognizing the superior calibration of pre-trained ancestors, we can develop post-hoc methods to "re-calibrate" helpful models without sacrificing their conversational utility. Future work should extend these insights to open-ended generation where logit extraction is more complex.
Summary updated