Rate-Oriented Point-Wise Confidence Bounds for ROC Curves

Louise A. C. Millard, Meelis Kull, Peter A. Flach

Abstract

Common approaches to generating confidence bounds around ROC curves have several shortcomings. We resolve these weaknesses with a new ‘rate-oriented’ approach. We generate confidence bounds composed of a series of confidence intervals for a consensus curve, each at a particular predicted positive rate (PPR), with the aim that each confidence interval contains new samples of this consensus curve with probability 95%. We propose two approaches; a parametric and a bootstrapping approach, which we base on a derivation from first principles. Our method is particularly appropriate with models used for a common type of task that we call rate-constrained, where a certain proportion of examples needs to be classified as positive by the model, such that the operating point will be set at a particular PPR value.
BibTeX
@inproceedings{millard2014rate,
  author = {Louise A. C. Millard and Meelis Kull and Peter A. Flach},
  title = {{Rate-Oriented Point-Wise Confidence Bounds for ROC Curves}},
  booktitle = {ECML/PKDD},
  year = {2014},
  doi = {10.1007/978-3-662-44851-9\_26},
}

Loading related content...

Figure