Abstract
Modern machine learning models often prioritize classification accuracy over probabilistic calibration, leading to predicted probabilities that do not reflect true empirical frequencies. We present Bayesian Binning into Quantiles (BBQ), a post-processing method that utilizes Bayesian model averaging over multiple binning schemes. BBQ produces more accurate and robust probability estimates than standard techniques like Platt scaling or isotonic regression. Our evaluation across 30 datasets demonstrates that BBQ significantly improves calibration metrics while preserving the original model's discriminative power.
Introduction
In many high-stakes applications, such as medical diagnosis or financial risk assessment, the raw "yes/no" output of a classifier is insufficient. Decision-makers require well-calibrated probabilities—estimates where a predicted probability of $p$ corresponds to the event occurring with frequency $p$ in the long run.
However, popular algorithms like Support Vector Machines (SVM) and Naive Bayes (NB) are often poorly calibrated. For instance, SVMs tend to produce scores that are pushed toward the extremes, while Naive Bayes often yields overconfident predictions due to its independence assumptions. Existing remedies, such as Platt scaling (parametric) and isotonic regression (non-parametric), have significant drawbacks: Platt scaling is restricted by its sigmoid assumption, and isotonic regression is prone to overfitting, especially in data-sparse regions.
Methods: Bayesian Binning into Quantiles (BBQ)
The proposed BBQ method addresses these issues by extending the traditional histogram binning approach. Histogram binning groups predictions into discrete bins and uses the observed frequency within each bin as the calibrated probability. The performance of this method is highly sensitive to the choice of the number of bins.
BBQ overcomes this sensitivity through two key innovations:
- Multiple Binning Configurations: Instead of selecting a single number of bins, BBQ considers a set of possible configurations (e.g., from 1 to $N$ bins).
- Bayesian Model Averaging: BBQ assigns a posterior probability to each binning configuration using the BDeu score, which measures how well the configuration explains the training data. The final calibrated probability is a weighted average across all configurations.
This Bayesian approach allows BBQ to be flexible in regions with abundant data while remaining conservative (averaging toward the prior) where data is sparse, providing a natural smoothing effect.
Experimental Evaluation
The researchers evaluated BBQ against histogram binning, Platt scaling, and isotonic regression using three base classifiers: Logistic Regression (LR), SVM, and Naive Bayes. The testbed included:
- Synthetic Data: A non-linearly separable dataset to test robustness against model mismatch.
- Real-World Benchmarks: 30 binary classification datasets from the UCI and LibSVM repositories.
Results
BBQ demonstrated superior performance across multiple dimensions:
- Calibration Accuracy: BBQ was found to be statistically superior to other methods in terms of Expected Calibration Error (ECE) and Maximum Calibration Error (MCE).
- Robustness: In simulations where the base model (e.g., a linear SVM) was a poor fit for the underlying data, BBQ significantly reduced calibration error (e.g., from 0.28 to 0.03).
- Preservation of Discrimination: A critical finding is that BBQ improves calibration without degrading the model's ability to rank instances. As shown in Figure 2, the Area Under the Curve (AUC) for BBQ is statistically indistinguishable from the base classifier across different models.
Conclusion
Bayesian Binning into Quantiles (BBQ) provides a robust, non-parametric framework for probability calibration. By leveraging Bayesian model averaging, it avoids the pitfalls of single-model selection and provides more stable estimates than traditional binning or regression techniques. While currently focused on binary classification, BBQ represents a significant step toward more reliable probabilistic machine learning. Future work will explore extensions to multi-class settings and theoretical bounds on its convergence rates.
Summary updated