The Alignment Calibration Trade-Off in Large Language Models

74 concepts53 papers

When a large language model (LLM) states an answer, how much should we trust it? Ideally, a model's confidence should match its accuracy: if it assigns an 80% probability to a given response, it should be correct roughly 80% of the time. This statistical harmony is known as calibration.

Base language models—those trained purely on next-token prediction—are surprisingly well-calibrated. However, to make these models useful and safe, we subject them to post-training alignment, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). While alignment makes models significantly more helpful and conversational, it introduces a severe side effect: it systematically destroys their calibration, leaving them dangerously overconfident.

Understanding why this calibration decay happens, and how we can fix it, is a major focus of current LLM reliability research.

The Natural Calibration of Base Models

To understand why alignment breaks calibration, we must first understand why base models possess it in the first place.

Base models are trained to minimize cross-entropy loss over massive text corpora. To achieve the absolute minimum loss, a model must learn to output token probabilities that perfectly mirror the true empirical distribution of human language. As demonstrated by Preetum Nakkiran et al., this token-level optimization gives rise to an emergent property called semantic calibration. Even though the model is only trained to predict the literal next word, its internal probabilities naturally reflect its true likelihood of being conceptually correct.

This calibration is not limited to text. In vision-language models (VLMs) like CLIP, the zero-shot state is similarly well-calibrated out of the box, as noted by Shuoyuan Wang et al.. When these models encounter uncertainty, their internal logit distributions smoothly flatten out to signal doubt.

Why Alignment Induces Overconfidence

Post-training alignment fundamentally alters this optimization landscape. Instead of predicting the natural distribution of human text, aligned models are trained to maximize human preference rewards or adhere strictly to instruction formats. This shift introduces two primary mechanisms that drive models toward extreme overconfidence.

1. The Assertiveness Premium

Human evaluators suffer from a distinct cognitive bias: they prefer answers that sound authoritative, structured, and helpful over responses that sound hesitant or uncertain. As Mark Steyvers et al. observe, RLHF training processes inadvertently capitalize on this bias. Because human judges reward assertive explanations, the optimization pipeline penalizes the model for expressing doubt, incentivizing it to adopt an authoritative tone even when its underlying factual certainty is low.

2. Probability Polarization

From a mathematical perspective, standard alignment objectives force a collapse in the model's output distribution. When a model is fine-tuned to output a specific preferred response format, the training objective aggressively drives the probability mass toward that single target sequence.

Eric Mitchell et al. and André F Cruz et al. show that this optimization causes probability polarization. Instead of maintaining a nuanced distribution over multiple potential answers, the model concentrates nearly 100% of its probability mass onto its preferred response. Leonidas Zotos et al. describe this as a flattening of the model's uncertainty profile, which erases the natural linguistic doubt that would otherwise correlate with task difficulty.

As a result, the model prioritizes the expected format and helpfulness of the response over an accurate representation of its internal uncertainty, as detailed by Benfeng Xu et al. and Yixin Liu et al.. The model becomes "confidently wrong," stating falsehoods or hallucinations with absolute certainty. This phenomenon extends across modalities; for instance, Shuoyuan Wang demonstrates that task-specific fine-tuning similarly wrecks the open-vocabulary calibration of vision models.

[Base Model: Balanced Distribution]
  - Correct Answer: 45%
  - Plausible Misconception: 35%
  - Unrelated Token: 20%
  (Nuanced, well-calibrated uncertainty)

       │ Post-Training Alignment (RLHF / SFT)
       ▼ 
[Aligned Model: Polarized Distribution]
  - Chosen Answer (Correct or Incorrect): 99.8%
  - All Other Options: 0.2%
  (Extreme overconfidence, destroyed calibration)

Mitigating the Confidence Gap

Because the internal token logits of aligned models are structurally distorted by polarization, standard calibration techniques like global temperature scaling often fail to generalize. Researchers have developed several distinct strategies to extract reliable confidence estimates from aligned models.

Bypassing Logits via Sample Consistency

If a model's raw token probabilities are no longer trustworthy, we can look at its behavior across multiple independent generation paths. By sampling from the model multiple times at a high temperature, we can measure how consistently it returns the same answer.

Niket Tandon et al. and Jeremy R. Cole et al. demonstrate that this "wisdom of the samples" approach serves as an exceptionally reliable confidence metric. While alignment skews individual token logits, it does not completely erase the model's underlying semantic consistency; if a model generates the exact same answer across ten distinct random paths, that answer is highly likely to be correct.

Eliciting Verbalized Confidence

Another surprising solution is to bypass the model's internal math entirely and simply ask it to state its confidence in plain text (e.g., "On a scale of 0-100, how sure are you?"). Eric Mitchell et al. show that this verbalized confidence can reduce calibration error by up to 50%. This suggests that while the alignment process ruins the model's raw token probabilities, the model's internal hidden states still retain a accurate representation of its own uncertainty, which can be extracted via natural language.

Re-Anchoring to the Pre-trained Base

Since pre-trained base models are naturally well-calibrated, several frameworks use the base model as an unsupervised anchor to correct its post-trained counterpart.

Methods like Disagreement-Aware Confidence Alignment (DACA) (detailed in Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator) and DUAL-ALIGN by Cheng Wang et al. operate by mapping the overconfident output distributions of an aligned model back onto the well-calibrated probability space of the original pre-trained base model. This allows the system to retain the conversational capabilities of the aligned model while restoring the statistical honesty of the base model.

Adaptive and Dynamic Scaling

Rather than applying a single, blunt temperature correction to all outputs, newer methods dynamically adjust calibration based on context. Johnathan Xie et al. introduced Adaptive Temperature Scaling (ATS), which predicts a unique temperature for every token by reading the model's internal hidden states. Similarly, the THERMOMETER framework by Maohao Shen et al. uses a lightweight auxiliary model to predict task-specific temperatures from unlabeled data, smoothing out polarized distributions on the fly.

Key Foundational Work

On the Calibration of Large Language Models and Alignment (2023) — contributed 4 concept elements to this direction.
A Study on the Calibration of In-context Learning (2024) — contributed 3 concept elements to this direction.
Are Large Language Models Bayesian? A Martingale Perspective on In-Context Learning (2024) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_alignment_induced_overconfidence_35b369fb,
  title = {alignment-induced overconfidence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/alignment-induced-overconfidence/rdr_35b369fbc1df41c78a1e6b53fc494974},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_alignment_induced_overconfidence_35b369fb_v1,
  title = {alignment-induced overconfidence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/alignment-induced-overconfidence/rdr_35b369fbc1df41c78a1e6b53fc494974/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake