Aligning Model Confidence via Proper Scoring Rules

53 concepts45 papers2 proposals

Modern deep learning models and large language models (LLMs) are incredibly capable, but they suffer from a dangerous flaw: they don't know what they don't know. When trained on standard accuracy-maximizing objectives or binary rewards (such as a simple "correct" or "incorrect" signal), a model is incentivized to become an overconfident guesser.

To understand why, consider a model evaluated purely on whether its top choice is correct. If the model believes an answer has a 51% chance of being right, its best strategy to maximize its expected reward under a binary evaluation is to assert that answer with 100% confidence. This mismatch between internal certainty and external expression leads to severe miscalibration, causing models to hallucinate or make high-confidence blunders in critical domains like medical diagnostics, forecasting, and autonomous tool use.

To build truly reliable systems, we must change the rules of the game. We need optimization objectives where the mathematically optimal strategy for the model is to be perfectly honest about its uncertainty. This is the domain of strictly proper scoring rules.

The Mechanics of Honesty

A scoring rule evaluates a predicted probability distribution against the actual ground-truth outcome. A scoring rule is defined as strictly proper if the expected reward is uniquely maximized (or the expected loss is uniquely minimized) when the predicted distribution exactly matches the true underlying probability distribution of the data.

To see why this forces honesty, let's look at two foundational proper scoring rules: the Brier Score and Logarithmic Loss.

The Brier Score

The Brier score is essentially the mean squared error applied to probabilities. For a binary event, if a model predicts a probability $p$ that the event will occur, and the true outcome is $y \in \{0, 1\}$, the Brier loss is:

$$\text{Loss} = (p - y)^2$$

Imagine a forecasting scenario where an event actually has a 70% chance of occurring ($P(y=1) = 0.7$). If the model honestly reports $p = 0.7$, its expected loss is:

$$0.7 \times (0.7 - 1)^2 + 0.3 \times (0.7 - 0)^2 = 0.21$$

What happens if the model tries to "game" the system by overconfidently claiming 100% certainty ($p = 1.0$)? Its expected loss becomes:

$$0.7 \times (1.0 - 1)^2 + 0.3 \times (1.0 - 0)^2 = 0.30$$

Because $0.30 > 0.21$, the overconfident strategy yields a worse score. The mathematical structure of the quadratic penalty ensures that any deviation from the true probability—whether overconfidence or underconfidence—incurs a higher expected loss.

Logarithmic Loss

Logarithmic loss (or negative log-likelihood) measures the log-probability assigned to the true outcome:

$$\text{Loss} = -\log(p_{\text{true}})$$

If a model assigns a 100% probability to an incorrect hypothesis, $p_{\text{true}}$ becomes 0, and the loss shoots to infinity. This severe penalty forces the model to spread its probability mass across multiple hypotheses whenever it faces genuine ambiguity, directly discouraging unwarranted certainty.

Incentivizing Doubt in Language Models

As language models transition from static text predictors to autonomous agents, integrating proper scoring rules into their training loop has become vital. Standard Reinforcement Learning from Human Feedback (RLHF) typically relies on binary or preference-based rewards, which inadvertently encourage models to output authoritative-sounding but incorrect answers.

To counteract this, Mehul Damani et al. introduced RLCR (Reinforcement Learning with Calibration Rewards) in Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty. By augmenting the standard reinforcement learning reward function with a Brier score, the authors penalized both incorrect answers and misaligned confidence levels, successfully training models to be honest about their own uncertainty.

A similar philosophy underpins the "Rewarding Doubt" framework proposed by David Bani-Harouni et al. in Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models. They utilized the logarithmic scoring rule as an RL reward function. Because the system rewards the log-probability of the actual outcome, the maximum expected reward is mathematically unattainable unless the model's reported confidence perfectly tracks its empirical accuracy.

This optimization strategy allows models to express calibration in different modalities:

  • Verbalized Confidence: Models can be trained to state their certainty in natural language (e.g., "I am 70% sure that..."). As shown by Neil Band et al. in Linguistic Calibration of Long-Form Generations, optimizing language models using proper scoring rules produces text that is significantly more honest about its knowledge gaps without sacrificing raw accuracy.
  • Numerical Forecasting: For explicit prediction tasks, Nikhil Chandak et al. demonstrated in Scaling Open-Ended Reasoning to Predict the Future that blending accuracy metrics with Brier scores enables open-ended reasoning models to rival professional human forecasters in predicting world events.
  • Explicit Confidence Scores: In Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning, Jiayun Wu et al. trained models to output a distinct numerical confidence score alongside their answers, using strictly proper scoring rules to ensure the score directly reflects the true probability of correctness.

Calibrating Continuous and Multivariate Predictions

While the Brier score is ideal for discrete or binary outcomes, many real-world problems require predicting continuous variables (like weather, financial trends, or time-to-event survival analysis). In these settings, minimizing standard Mean Squared Error (MSE) only forces the model to predict the mean of the distribution, providing no incentive to accurately map out the range of uncertainty.

To solve this, researchers adapt proper scoring rules to continuous distributions using the Continuous Ranked Probability Score (CRPS). CRPS generalizes the Brier score by evaluating the entire cumulative distribution function (CDF) against a step function representing the true outcome.

In survival analysis, where data is often "censored" (e.g., a patient survives past the end of a study, so their exact event time is unknown), standard maximum likelihood estimation can over-penalize confident mistakes. As explored in Countdown Regression: Sharp and Calibrated Survival Predictions, switching the training objective to a robust survival-adapted CRPS forces models to produce "sharper" (narrower, more useful) prediction intervals while maintaining strict probabilistic calibration.

When moving to multivariate regression—where a model must predict multiple correlated continuous variables simultaneously—calculating proper scores becomes computationally challenging. The authors of Scoring rule nets: beyond mean target prediction in multivariate regression addressed this by developing Conditional CRPS (CCRPS), ensuring that neural networks converge to global optima where both the individual predictions and their predicted correlations perfectly match the ground truth. To handle outliers in complex spatio-temporal forecasting, Lijun Sun et al. introduced MVG-CRPS: A Robust Loss Function for Multivariate Probabilistic Forecasting, leveraging a whitening transformation to maintain the theoretical guarantees of strictly proper scoring rules with high computational efficiency.

Entropy Constraints and Robustness

Beyond serving as direct loss functions, proper scoring rules provide a framework for understanding and regularizing model behavior under distribution shifts. When a model encounters out-of-distribution (OOD) data, its internal representations often degrade, yet its outputs remain dangerously overconfident.

One way to combat this is through entropy-constrained calibration. Minimizing negative log-likelihood is mathematically tied to maximizing the entropy of the output distribution under empirical constraints. By explicitly forcing higher entropy on incorrect or uncertain predictions, models can be prevented from collapsing into overconfidence.

For instance, Dexter Neo et al. introduced MaxEnt Loss: Constrained Maximum Entropy for Calibration under Out-of-Distribution Shift, which uses statistical constraints based on the Principle of Maximum Entropy to maintain stable calibration even when the testing data looks radically different from the training data.

Proper scores also offer a more robust foundation for downstream tasks like active learning. Traditional active learning algorithms often select new data points to label based on simple classification error or heuristic uncertainty. However, David Tan et al. demonstrated in Diversity Enhanced Active Learning with Strictly Proper Scoring Rules that constructing acquisition functions out of strictly proper scoring rules (like the Brier score or log-likelihood) leads to far more effective sample selection. Because proper scores incentivize the model to accurately map its entire probability distribution, the resulting uncertainty estimates are highly reliable, allowing the active learning loop to pinpoint the exact data points that will most effectively improve the model's knowledge.

By shifting our optimization objectives away from brittle, binary accuracy metrics and toward strictly proper scoring rules, we fundamentally alter how models learn. Instead of rewarding models for making lucky guesses, proper scoring rules make truthfulness and self-awareness the most profitable strategy.

Key Foundational Work

Evaluating semantic-level confidence scores with multiple hypotheses (2008) — contributed 1 concept elements to this direction.
Exploring Prediction Uncertainty in Machine Translation Quality Estimation (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_proper_scoring_rule_optimization_b61887a3,
  title = {proper scoring rule optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/proper-scoring-rule-optimization/rdr_b61887a395d14ab2894475a20cd05787},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_proper_scoring_rule_optimization_b61887a3_v1,
  title = {proper scoring rule optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/proper-scoring-rule-optimization/rdr_b61887a395d14ab2894475a20cd05787/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake