Making learning more transparent using conformalized performance prediction

Matthew J. HollandarXiv 2020 · arXiv

Abstract

Conformal Performance Prediction (CPP) provides a distribution-free framework for generating well-calibrated predictive intervals for machine learning performance metrics. Unlike traditional PAC (Probably Approximately Correct) bounds, which are often too pessimistic for practical use, CPP offers computable guarantees on future loss. This paper details the CPP methodology, distinguishing between model-specific (Candidate) and algorithm-wide (Algorithm) predictions, and demonstrates its utility across classification and regression tasks.

Introduction

A fundamental question in machine learning is: "How well will my model perform on the next unseen example?" Standard statistical learning theory typically addresses this using PAC bounds. However, these bounds are often distribution-dependent or so loose that they lack practical utility for real-world decision-making.

This paper introduces Conformal Performance Prediction (CPP), which adapts the principles of conformal prediction—traditionally used for label uncertainty—to the prediction of performance metrics like loss or error rates. CPP allows users to specify a confidence level (e.g., 90%) and receive a range of expected performance that is guaranteed to be well-calibrated under the assumption of data exchangeability.

The CPP Framework

The CPP framework builds on "Split Conformal Prediction" to create two primary classes of prediction:

  1. Candidate CPP: Focuses on a fixed, pre-trained model. It utilizes a calibration set to construct an interval that covers the loss the model will incur on a single new data point.
  2. Algorithm CPP: Accounts for uncertainty in the learning process itself. It predicts the loss an algorithm will incur after being trained on a fresh training set and evaluated on a new data point. This includes:
  3. Z-free CPP: Provides a general performance interval for the algorithm.
  4. Z-modulated CPP: Uses regression to provide instance-specific performance intervals.

The framework is "distribution-free," requiring only that the data be exchangeable (a property satisfied by i.i.d. data).

Experimental Evaluation

The CPP framework was evaluated using two distinct setups:

  • Digits Classification: Using the UCI Optical Recognition of Handwritten Digits dataset, we compared Linear and RBF-kernel SVMs.
  • Simulated Linear Regression: A synthetic dataset with additive noise (Normal and Student-t distributions) was used to compare Batch Gradient Descent (BGD) and Stochastic Gradient Descent (SGD) under heavy-tailed noise conditions.

Results and Discussion

SVM Performance and Transparency

On the Digits dataset, CPP provides a transparent comparison between model architectures. As shown in Figure 1, the RBF-kernel SVM (blue) exhibits significantly tighter and more stable CPP intervals compared to the linear SVM (magenta). This indicates not only a lower average loss but also higher reliability in its performance predictions.

CPP width distributions and performance intervals for SVM-rbf and SVM-lin on the Digits dataset.

Optimization and Robustness

In the comparison between BGD and SGD, CPP intervals successfully captured the variance inherent in different optimization methods. Figure 2 illustrates how the distribution of CPP widths varies between GD and SGD across different noise regimes. In scenarios with heavy-tailed noise, the "Algorithm CPP" approach allows researchers to quantify the robustness of the learning procedure across many potential training realizations.

Comparison of CPP width distributions for Gradient Descent (GD) and Stochastic Gradient Descent (SGD) under different noise regimes.

Conclusion

CPP offers a rigorous yet practical method for predicting machine learning performance. By providing well-calibrated intervals rather than single-point estimates or loose bounds, it enhances the transparency of AI systems. Future work will focus on improving data efficiency and extending the framework to handle covariate shift, where the data distribution changes between training and deployment.

Summary updated

Abstract

In this work, we study some novel applications of conformal inference techniques to the problem of providing machine learning procedures with more transparent, accurate, and practical performance guarantees. We provide a natural extension of the traditional conformal prediction framework, done in such a way that we can make valid and well-calibrated predictive statements about the future performance of arbitrary learning algorithms, when passed an as-yet unseen training set. In addition, we include some nascent empirical examples to illustrate potential applications.
BibTeX
@misc{holland2020making,
  author = {Matthew J. Holland},
  title = {{Making learning more transparent using conformalized performance prediction}},
  year = {2020},
  eprint = {2007.04486},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.04486},
}
Figure