Confidence Bands for ROC Curves

Sofus A. Macskassy, Foster Provost, Michael L. Littman2003 · DOI

TL;DR

This paper tackles the challenge of reliably comparing machine learning classifiers by introducing and evaluating methods for generating "confidence bands" around ROC curves. They propose a novel "radial sweep" method and an optimization framework, demonstrating that these new approaches provide a more accurate and realistic assessment of classifier performance compared to prior techniques.

What Problem Does This Paper Address?

When comparing machine learning classifiers, especially in areas like diagnostics or information retrieval, Receiver Operating Characteristic (ROC) curves are a standard tool. An ROC curve plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various classification thresholds. While comparing the Area Under the Curve (AUC) is common, it can be too coarse. A more nuanced comparison involves understanding the uncertainty around an ROC curve, which is where confidence bands come in. The problem is that existing methods for generating these bands often produce intervals that are too "tight" (too narrow), failing to accurately reflect the true variability of classifier performance. This can lead to incorrect conclusions about which classifier is truly better.

Methods

The authors explore and introduce several methods for generating confidence bands for ROC curves:

  • Adaptation of Prior Methods: They adapt two existing techniques:
  • Vertical Sweep: This method sweeps a vertical line across the ROC space (from FPR=0 to 1) and generates confidence intervals for the True Positive Rate (TPR) at fixed FPR points.
  • Threshold Sweep: This method sweeps across the possible thresholds of the scoring function and generates confidence intervals for both FPR and TPR at each threshold.
  • New Radial Sweep Method: This is the paper's novel contribution. Instead of sweeping vertically or horizontally, it sweeps radially from the origin (0,0) towards (1,1). At fixed angular intervals, it samples points where ROC curves intersect the sweep vector and generates confidence intervals based on the distance from the lower-right corner (1,0). This method aims to better account for the curvature of ROC curves.
  • Optimization Methodology: Recognizing that even their new methods might produce bands that are too tight, they introduce a general optimization framework. This involves generating an empirical distribution of ROC curves (e.g., via bootstrapping), creating initial bands using a chosen method (like radial sweep), and then iteratively adjusting the band boundaries to maximize the containment of the "fitting" curves while still aiming for a desired confidence level (e.g., 95%).
  • Evaluation Framework: They also propose a new way to evaluate confidence bands. Instead of strictly requiring an entire ROC curve to fall within the bands, they introduce a measure based on the percentage of points on an ROC curve that fall outside the bands. This allows for a more nuanced assessment of how well bands fit the data, especially when curves are highly variable.

Datasets

The primary dataset used for evaluation is the Covertype dataset from the UCI repository. To create a suitable scenario for ROC analysis, they focused on the two most frequent classes within this dataset, resulting in a dataset of 495,141 instances. They used various training and test set sizes, as well as bootstrapping techniques, to generate distributions of ROC curves for evaluation.

Key Insights and Ideas

  • Prior methods are too tight: Existing techniques for generating confidence bands on ROC curves often underestimate the true variability, leading to bands that are too narrow and fail to contain the expected proportion of ROC curves.
  • Radial sweep offers a better approximation: The novel radial sweep method provides a more intuitive and empirically better-performing way to approximate confidence bands that are more "normal" to the ROC curve's tangent, accounting for curvature more effectively than purely vertical or horizontal sweeps.
  • Optimization is crucial: Even with a better sweep method, optimization is necessary to achieve bands that truly reflect the data's variability. Their simple optimization approach significantly improves band containment.
  • New evaluation metric is needed: The traditional strict containment requirement for confidence intervals is too demanding for the often chaotic nature of ROC curves. A measure based on the percentage of points outside the bands provides a more realistic evaluation of band fit.
  • Empirical distribution is key: Using an empirical distribution (derived from bootstrapping) for generating confidence intervals and bands generally leads to better results than assuming normal or binomial distributions, especially for the radial sweep method.

Key Results

  • Improved Containment: The optimized radial sweep method significantly improved the percentage of ROC curves contained within the confidence bands compared to non-optimized methods. For the "fitting" data used to generate the bands, they achieved 96.8% containment.
  • Test Set Performance: While the optimized method performed better on the test set than prior methods, it still fell short of the ideal 95% containment, achieving 86.2% with the radial sweep and empirical distribution.
  • New Evaluation Metric Effectiveness: Using their new evaluation metric (epsilon, ε), they showed that the optimized radial sweep method required a much smaller ε (0.02) to achieve 95% containment on the test set compared to other methods (e.g., vertical sweep required 0.5417). This indicates a better fit.
  • Radial Sweep Outperforms Others: Across various distribution assumptions and evaluation metrics, the radial sweep method generally outperformed vertical and threshold sweep methods.

Limitations and Future Work

  • Radial Sweep Approximation: While the radial sweep method is an improvement, it's still an approximation of bands that are truly normal to the ROC curve. A more sophisticated approach might yield better results.
  • Optimization Simplicity: The optimization method used was simple. More complex optimization strategies could potentially lead to even better-fitting bands.
  • Sampling Density: The paper did not deeply investigate the sensitivity of the bands to the number of points sampled along the sweep.
  • Band Tightness/Looseness: It remains an open question whether the optimized bands are uniformly tight or loose across all regions of the ROC curve.
  • Chaotic Curves: The inherent crisscrossing nature of some ROC curves makes it challenging to create bands that perfectly capture expected containment. Future work could explore how to handle such chaotic behavior more effectively.

Summary updated

Abstract

We address the problem of comparing the performance of classifiers. In this paper we study techniques for generating and evaluating confidence bands on ROC curves. Historically this has been done using one-dimensional confidence intervals by freezing one variable-the false-positive rate, or threshold on the classification scoring function. We adapt two prior methods and introduce a new radial sweep method to generate confidence bands. We show, through empirical studies, that the bands are too tight and introduce a general optimization methodology for creating bands that better fit the data, as well as methods for evaluating confidence bands. We show empirically that the optimized confidence bands fit much better and that, using our new evaluation method, it is possible to gauge the relative fit of different confidence bands.
BibTeX
@misc{macskassy2003confidence,
  author = {Sofus A. Macskassy and Foster Provost and Michael L. Littman},
  title = {{Confidence Bands for ROC Curves}},
  year = {2003},
  doi = {10.21236/ada453849},
}
Figure