Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers

Meelis Kull, Telmo de Menezes e Silva Filho, Peter A. FlachAISTATS 2017 · OpenReview

Abstract

For optimal decision making under variable class distributions and misclassification costs, a classifier needs to produce well-calibrated estimates of the posterior probability. Isotonic calibration is a powerful non-parametric method, but it is prone to overfitting. Logistic calibration (Platt scaling) is a popular parametric alternative, but it makes strong assumptions about the distribution of classifier scores. We propose Beta calibration, a new parametric method derived from the assumption that scores follow a Beta distribution. Beta calibration is more flexible than logistic calibration, includes the identity map, and can be easily implemented using standard logistic regression tools.

Introduction: The Calibration Problem

In machine learning, we often require more than a discrete class prediction; we need to know the model's certainty. A well-calibrated model is one where a predicted probability of 0.8 corresponds to an 80% frequency of the positive class in reality. However, many popular algorithms—like Naive Bayes and AdaBoost—are notoriously poorly calibrated, often producing overconfident or skewed probabilities.

To address this, researchers typically use post-hoc calibration methods. The two most common are Isotonic Calibration, which is non-parametric and flexible but prone to overfitting, and Logistic Calibration (Platt scaling), which is parametric but limited by its underlying assumptions.

The Limitations of Logistic Calibration

Logistic calibration assumes that the scores for each class follow a Normal (Gaussian) distribution. This leads to two primary issues:

  1. Distribution Mismatch: Classifier scores are usually bounded on a [0, 1] scale and often exhibit heavy skews, making the infinite-range Gaussian assumption a poor fit.
  2. The Identity Crisis: Mathematically, the logistic curve family does not include the identity function ($f(s) = s$). This means logistic calibration can actually degrade the calibration of a model that was already performing well.
The family of logistic calibration maps, illustrating their inability to represent the identity function.

Beta Calibration: A Well-Founded Alternative

We propose Beta Calibration, derived from the assumption that scores follow a Beta distribution. Because the Beta distribution is defined on the interval [0, 1], it is a much more natural fit for probability scores than the Gaussian distribution.

The resulting 3-parameter calibration map is defined as: $$\mu_{beta}(s; a, b, c) = \frac{1}{1 + \frac{1}{e^c \frac{s^a}{(1-s)^b}}}$$

Implementation via Logistic Regression

One of the most practical aspects of Beta calibration is its ease of implementation. Fitting the model is mathematically equivalent to performing a bivariate logistic regression. By using $\ln(s)$ and $-\ln(1-s)$ as features, practitioners can implement Beta calibration using standard machine learning libraries like scikit-learn in just a few lines of code.

Key Insights and Empirical Results

Beta calibration offers several advantages over existing methods:

  • The Identity Map: Unlike logistic calibration, Beta calibration can represent the identity function (when $a=b=1$ and $c=0$), ensuring it won't break a good model.
  • Flexibility: It can model "inverse-sigmoid" shapes and handle asymmetric distributions where positive and negative scores behave differently.

We conducted an extensive study using 41 UCI benchmark datasets and base classifiers known for poor calibration (Naive Bayes and AdaBoost). Performance was evaluated using Log-loss and Brier Score.

Comparison of Beta calibration (red) against sigmoid (blue) and isotonic (green) methods on benchmark datasets.

Our results show that Beta calibration is significantly better than logistic calibration for Naive Bayes and AdaBoost. It also proves more robust than isotonic calibration on smaller datasets where overfitting is a concern.

Conclusion

Beta calibration provides a theoretically sound and practically superior alternative to the industry-standard Platt scaling. By adopting a more appropriate distributional assumption, it offers the flexibility of non-parametric methods with the robustness and simplicity of parametric ones. We recommend it as a standard tool for any machine learning pipeline requiring accurate probability estimates.

Summary updated

Abstract

For optimal decision making under variable class distributions and misclassification costs a classifier needs to produce well-calibrated estimates of the posterior probability. Isotonic calibration...
BibTeX
@inproceedings{kull2017beta,
  author = {Meelis Kull and Telmo de Menezes e Silva Filho and Peter A. Flach},
  title = {{Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2017},
  url = {https://openreview.net/forum?id=AJfz81Aycr},
}
Figure