Pointwise sampling uncertainties on the Precision-Recall curve

Ralph E. Q. Urlus, Max Baak, Stéphane Collot, Ilan Fridman RojasAISTATS 2023

Abstract

Precision-Recall (PR) and Receiver Operating Characteristic (ROC) curves are standard tools for evaluating binary classifiers. However, they are typically presented as single lines without accounting for sampling uncertainty. This paper introduces a computationally efficient method based on Wilks’ theorem to generate pointwise uncertainty contours and continuous bands for these curves. The proposed approach is significantly faster than bootstrapping and maintains high accuracy even for small datasets.

Introduction

In machine learning evaluation, the finite size of test sets introduces sampling noise. A different sample of the same population would yield a different PR or ROC curve. Traditional methods like bootstrapping to estimate this uncertainty are computationally prohibitive for large-scale evaluations and can produce artifacts in data-scarce regions. This work addresses the need for a fast, robust statistical framework to quantify uncertainty in model performance metrics.

Methodology

The authors model the confusion matrix—comprising True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN)—as a multinomial distribution. They evaluate four primary methods for deriving joint uncertainty of precision and recall:

  1. Frequentist (Profile Likelihood): An exact but slow baseline using Monte Carlo simulations.
  2. Wilks’ Method: Approximates the log-likelihood ratio using a $\chi^2$ distribution, offering a closed-form-like speed without simulation.
  3. Bayesian Approach: Utilizes a Dirichlet conjugate prior to generate a posterior distribution.
  4. Bivariate Normal Approximation: A fast linear approximation that often fails near the boundaries (precision/recall $\approx$ 0 or 1).
Precision-Recall curve with pointwise uncertainty contours and continuous uncertainty bands.

Experimental Setup

The methods were validated using synthetic datasets generated via scikit-learn's make_blobs. The study performed extensive coverage analysis across test set sizes ranging from $10^1$ to $10^5$ records. By using synthetic data, the "true" population parameters were known, allowing for precise measurement of statistical coverage—the frequency with which the calculated intervals contain the true value.

Results

Wilks’ method emerged as the superior approach, balancing speed and statistical rigor. It is approximately 400,000 times faster than the frequentist baseline while providing nearly identical results.

Statistical coverage comparison between Wilks’ method and Bivariate Normal approximation across varying test set sizes.

Key findings include:

  • Small Sample Robustness: Wilks’ method remains accurate for samples as small as 10 records, whereas the Bivariate Normal approximation significantly underestimates uncertainty for sets smaller than 10,000 points.
  • Computational Efficiency: Calculating a full uncertainty band takes ~242 ms with Wilks’ method, compared to 7.5 hours for the Frequentist approach.
  • Boundary Accuracy: Unlike linear approximations, Wilks’ method correctly handles the non-elliptical geometry of uncertainty near the edges of the PR space.

Conclusion

The application of Wilks’ theorem to the multinomial distribution of confusion matrices provides a practical solution for adding uncertainty bands to performance curves. The authors have released the MMU (Model Metric Uncertainty) Python library to facilitate the adoption of these methods in standard machine learning workflows. Future work may extend this framework to multi-class classification and account for other sources of uncertainty, such as label noise.

Summary updated

BibTeX
@inproceedings{urlus2023pointwise,
  author = {Ralph E. Q. Urlus and Max Baak and Stéphane Collot and Ilan Fridman Rojas},
  title = {{Pointwise sampling uncertainties on the Precision-Recall curve}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
}
Figure