Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification

Yu Bai, Song Mei, Huan Wang, Caiming Xiong

Introduction

In modern machine learning, high-accuracy models are frequently plagued by miscalibration. Specifically, they tend to be over-confident: the predicted probability of a class often exceeds the actual frequency of that class in the data. While the prevailing consensus attributes this phenomenon to over-parametrization—where massive models "memorize" training data—this paper demonstrates that over-confidence is a fundamental property of even simple, under-parametrized models like logistic regression.

The Calibration Problem

Calibration measures the alignment between a model's predicted confidence and its empirical accuracy. A perfectly calibrated model that predicts 90% confidence for a set of samples should be correct 90% of the time. In deep learning, over-confidence is often blamed on the model's capacity to push weights to extremes to minimize training loss.

Reliability diagram for a WideResNet on ImageNet, showing the typical over-confidence observed in deep learning models where confidence (red) exceeds accuracy (blue).

However, the authors investigate whether a "well-specified" model—where the architecture matches the data-generating process and data is abundant ($n \gg d$)—can still be systematically miscalibrated.

Theoretical Framework

The researchers analyze Empirical Risk Minimization (ERM) in a high-dimensional "proportional limit," where both the number of features ($d$) and the number of samples ($n$) grow large while their ratio ($d/n$) remains constant.

By focusing on a "realizable" scenario (data generated by a linear model with a specific activation function), they utilize advances in high-dimensional statistics to characterize the solution of logistic regression through a system of nonlinear equations. A local linear analysis for small $d/n$ ratios allows them to derive a closed-form expression for calibration error, revealing that the mathematical structure of the activation function itself introduces bias.

Key Insights: Curvature and Scaling

The study reveals several critical findings regarding the nature of model bias:

  • Inherent Bias in Logistic Regression: Logistic regression is shown to be inherently over-confident in high-dimensional regimes, regardless of whether the model has "memorized" the data.
  • The Role of Concavity: The over-confidence is driven by the concavity of the activation function (e.g., the sigmoid function) on the positive domain ($z > 0$). This geometric property causes the model's average prediction to be higher than the true underlying probability.
  • The $d/n$ Scaling Law: Calibration error is proportional to the ratio of parameters to data points ($\Theta(d/n)$). As features are added, the model becomes more over-confident even before reaching the over-parametrized regime.
  • Under-confidence is Possible: Surprisingly, over-confidence is not a universal law. By designing a specific "wavy" activation function that is convex in certain regions, the authors successfully created a model that is systematically under-confident.
Reliability diagram for realizable logistic regression, demonstrating that over-confidence is inherent even in simple, well-specified models.

Experimental Results

The authors validated their theory using synthetic Gaussian data, CIFAR-10 (with pseudo-labels to create a realizable environment), and ImageNet.

  • Simulation Match: Theoretical "analytical" curves for calibration matched experimental simulations almost perfectly.
  • Real-World Generalization: Experiments on CIFAR-10 confirmed that the over-confidence persists even when the Gaussian input assumption is relaxed, suggesting the theory applies to complex natural image distributions.
  • Calibration Error Magnitude: The study found that the overall Calibration Error (CE) scales as $O(d/n)$, providing a much tighter characterization than previous bounds.

Conclusion and Future Work

This work provides a precise characterization of calibration in binary classification, proving that over-confidence is not merely a byproduct of model size but an intrinsic feature of the learning algorithm and activation function. While the current proof relies on Gaussian assumptions and binary classification, the empirical results suggest broader applicability. Future research may explore the interplay between this intrinsic bias and common regularization techniques like label smoothing or weight decay.

Summary updated

Abstract

Modern machine learning models with high accuracy are often miscalibrated -- the predicted top probability does not reflect the actual accuracy, and tends to be over-confident. It is commonly believed that such over-confidence is mainly due to over-parametrization, in particular when the model is large enough to memorize the training data and maximize the confidence. In this paper, we show theoretically that over-parametrization is not the only reason for over-confidence. We prove that logistic regression is inherently over-confident, in the realizable, under-parametrized setting where the data is generated from the logistic model, and the sample size is much larger than the number of parameters. Further, this over-confidence happens for general well-specified binary classification problems as long as the activation is symmetric and concave on the positive part. Perhaps surprisingly, we also show that over-confidence is not always the case -- there exists another activation function (and a suitable loss function) under which the learned classifier is under-confident at some probability values. Overall, our theory provides a precise characterization of calibration in realizable binary classification, which we verify on simulations and real data experiments.
BibTeX
@inproceedings{bai2021don,
  author = {Yu Bai and Song Mei and Huan Wang and Caiming Xiong},
  title = {{Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=3mWjgi0mN3I},
}
Figure