Designing Classification Losses via Probability Elicitation

22 concepts15 papers3 proposals

When training a machine learning model for classification, we often require more than a simple binary or multiclass label. We want the model to output calibrated probabilities—reliable confidence scores that reflect the true likelihood of an event.

Historically, loss functions were chosen somewhat ad-hoc based on computational convenience, such as the hinge loss for Support Vector Machines or the exponential loss for boosting. However, these functions do not inherently incentivize a model to learn true probabilities.

The framework of probability elicitation flips loss function design on its head. Instead of picking a loss function and analyzing its behavior, we start with the statistical properties we want the model's predictions to satisfy, and systematically derive the exact loss function required to enforce them.

The Principle of Honest Incentives: Proper Scoring Rules

At the core of probability elicitation is the concept of a proper scoring rule. A scoring rule evaluates a predicted probability distribution against an actual observed outcome. It is considered proper if the expected loss is minimized when the model's predicted probability matches the true underlying probability of the data.

Predd et al. provided a foundational mathematical vindication of this concept, proving that a probabilistic forecast is coherent (consistent with the axioms of probability) if and only if it cannot be strictly beaten by a rival forecast across all outcomes under a proper scoring rule. This establishes that properness is not just a desirable feature, but an intrinsic requirement for valid probabilistic reasoning.

For a loss function to be useful for probability estimation, it must be strictly proper, meaning the true probability distribution is the unique minimizer of the expected loss. As Mey et al. and Loog demonstrate, non-strictly proper losses fail this criterion because they allow multiple distinct minimizers that do not reflect the true distribution once a basic classification threshold is satisfied.

When a loss function lacks strict properness, models have no mathematical incentive to learn true probabilities, often resulting in systematic overconfidence or underconfidence—a phenomenon Cheng et al. and Vasconcelos identify as a major flaw in many standard asymmetric losses used for imbalanced data.

The Geometry of Loss: Bayes Risk and Bregman Divergences

How do we systematically construct a strictly proper loss function? The answer lies in a beautiful geometric duality between Bayes risk and Bregman divergences.

Imagine a function that maps a true probability distribution to the minimal possible expected loss a classifier can achieve on that distribution. This is the Bayes risk function. By definition, the Bayes risk must be concave; as uncertainty increases (e.g., a 50/50 coin flip), the minimal achievable risk increases.

Expected Loss
     ^
     |       /---\       <- Concave Bayes Risk H(p)
     |      /     \
     |     /   *   \     <- Tangent plane at predicted q
     |    /   / \   \
     |   /   /   \   \
     +------------------> True Probability (p)
             ^   ^
             q   p       |<- Bregman Divergence = Loss Penalty ->|

As shown by Masnadi-Shirazi et al. (2008) and Abernethy et al., minimizing classification risk is mathematically identical to minimizing a Bregman divergence. The design recipe works as follows:

  1. Choose a strictly concave surface to represent your desired minimum conditional Bayes risk.
  2. Construct a tangent plane to this surface at your model's predicted probability $q$.
  3. Measure the vertical distance between the true surface at $p$ and the tangent plane.

This distance is the Bregman divergence. Because the surface is concave, the tangent plane always lies above the surface. The distance is exactly zero if and only if $q = p$. Thus, the Bregman divergence acts as a penalty that naturally forms a strictly proper scoring rule.

For multiclass problems, Vernet et al. extended this framework, proving that all proper multiclass losses are uniquely characterized by calculating the divergence associated with the slope of a chosen concave surface over a probability simplex.

Bridging Scores and Probabilities: Proper Composite Losses

In practice, neural networks and linear models rarely output raw probabilities directly. Instead, they output unconstrained real-valued scores, or logits. To map these raw scores back to the $[0, 1]$ probability interval, we apply a link function $\psi$ (such as the sigmoid or softmax function).

When we combine a loss function with a link function, we create what is known as a composite loss.

  Raw Model Score (f)  --->  [ Link Function ψ ]  --->  Probability Estimate (q)
                                                                |
                                                       [ Proper Scoring Rule ]
                                                                |
                                                                v
                                                           Loss Value

Reid et al. provided a complete mathematical characterization of proper composite losses. They proved that for a composite loss to remain convex—and thus computationally efficient to optimize—there must be a strict, matching harmony between the loss's internal weight function (which dictates sensitivity at different probability thresholds) and the derivative of the link function.

Why This Works: Applications and Implications

Framing loss design through probability elicitation solves several long-standing practical challenges in machine learning.

1. Cost-Sensitive Learning and Quantile Estimation

Standard classifiers treat all errors equally. However, in medical diagnosis or fraud detection, false negatives are far more costly than false positives.

By redesigning the classic hinge loss from the ground up using probability elicitation, Masnadi-Shirazi et al. (2010) derived a cost-sensitive SVM that remains theoretically consistent with the Bayes-optimal decision rule under arbitrary cost constraints. Similarly, Yu et al. demonstrated that minimizing an asymmetric hinge loss is mathematically equivalent to computing the entire quantile path of a conditional distribution, allowing models to serve as tools for probing underlying distributions rather than just outputting static labels.

2. Optimizing Complex Downstream Metrics

For complex performance metrics like the F-measure or Area Under the ROC Curve (AUC), the ideal decision boundary changes depending on the data distribution.

Dembczynski et al. and Agarwal et al. proved that minimizing a strongly proper surrogate loss (where the Bayes risk is strongly concave) guarantees that the model accurately recovers the true underlying probabilities. Once these true probabilities are recovered, users can flexibly tune decision thresholds on validation data to optimize complex metrics perfectly, strictly bounding the downstream surrogate regret.

3. Unifying Boosting and "Improper" Losses

Probability elicitation also bridges algorithmic frameworks like boosting with statistical consistency. Nock et al. (2008) and Nock et al. (2012) introduced balanced convex surrogates to ensure that boosting algorithms converge to true posteriors without overfitting confidence scores.

Remarkably, this perspective can even rescue losses traditionally labeled as "improper." Bao et al. demonstrated that many popular losses, such as the Focal Loss, are actually strictly proper composite losses in disguise. They introduce a "calmness" condition and a mathematical tool called the $\Psi$-transform, which allows practitioners to apply an inverse transformation to the model's raw outputs and successfully recover true, well-calibrated probabilities.

Key Foundational Work

On the Design of Loss Functions for Classification: theory, robustness to outliers, and SavageBoost (2008) — contributed 3 concept elements to this direction.
On the Efficient Minimization of Classification Calibrated Surrogates (2008) — contributed 1 concept elements to this direction.
Probabilistic coherence and proper scoring rules (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_probability_elicitation_loss_design_957c8657,
  title = {probability elicitation loss design},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/probability-elicitation-loss-design/rdr_957c8657f5d0441ba67b9f3139eef642},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_probability_elicitation_loss_design_957c8657_v1,
  title = {probability elicitation loss design},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/probability-elicitation-loss-design/rdr_957c8657f5d0441ba67b9f3139eef642/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake