Precision-Recall-Gain Curves: PR Analysis Done Right

Peter A. Flach, Meelis KullNeurIPS 2015 · OpenReview

TL;DR

Traditional Precision-Recall (PR) curves are mathematically "broken" because they use an arithmetic scale to evaluate $F$-scores, which are based on harmonic means. This paper introduces Precision-Recall-Gain (PRG) curves, a new coordinate system that fixes these inconsistencies, allows for linear interpolation, and provides an Area Under the Curve (AUPRG) that actually relates to expected $F_1$ scores.

The Problem with Traditional PR Curves

In binary classification—especially in fields like information retrieval or medical diagnosis—we often deal with "imbalanced" data where negative examples far outnumber positives. In these cases, standard accuracy is a poor metric, and Receiver Operating Characteristic (ROC) curves can be overly optimistic because they account for True Negatives.

Researchers often turn to Precision-Recall (PR) curves instead. However, the authors argue that PR analysis is "fraught with difficulties." Specifically:

  1. Incoherent Scales: We evaluate models using the $F_1$ score (a harmonic mean), but PR curves use arithmetic interpolation between points.
  2. Non-Universal Baselines: The "random" baseline on a PR plot changes depending on the class distribution, making it hard to compare across datasets.
  3. Uninterpretable Area: The Area Under the PR Curve (AUPR) doesn't have a clear functional meaning relative to standard performance metrics.

Precision-Recall-Gain (PRG) Curves

The authors propose a change of coordinates from $(Recall, Precision)$ to $(Recall\ Gain, Precision\ Gain)$.

The "Gain" transformation rescales the metrics from the interval $[\pi, 1]$ (where $\pi$ is the proportion of positives) to $[0, 1]$ using a harmonic scale. The formulas are:

  • Precision Gain ($precG$): $1 - \frac{\pi}{1-\pi} \cdot \frac{FP}{TP}$
  • Recall Gain ($recG$): $1 - \frac{\pi}{1-\pi} \cdot \frac{FN}{TP}$
Comparison of traditional PR and PRG curves

By moving to this "Gain" space, several mathematical properties align correctly:

  • Linear Interpolation: A straight line between two points in PRG space represents a realizable weighted combination of those two classifiers. In traditional PR space, this interpolation is incorrectly assumed to be linear.
  • $F_\beta$ Isometrics are Linear: Lines of constant $F_\beta$ score become straight lines with a slope of $-\beta^2$, whereas they are non-linear curves in standard PR space.
  • Convex Hull Optimality: Just like in ROC analysis, you can construct a convex hull. Every point on this hull is optimal for some range of $\beta$ values, allowing for easy model calibration.

Key Insights

  • Ignoring the "Bad" Zone: Any model with precision or recall lower than the class prior ($\pi$) is performing worse than an "always positive" classifier. PRG curves naturally ignore these regions by assigning them negative gain, focusing only on the "useful" part of the performance space.
  • Meaningful Area: The Area Under the PRG Curve (AUPRG) is shown to be a linear version of the expected $F_1$ score. This gives the area a formal justification that AUPR lacks.

Empirical Validation

To prove that this isn't just a theoretical exercise, the authors conducted a massive empirical study using the OpenML platform, evaluating nearly 26,000 models across 886 binary classification tasks.

Key Results: * Model Selection Disagreement: In 24% of the tasks, AUPR and AUPRG disagreed on which model was the "best." * Top-3 Disagreement: In 58% of the tasks, the two metrics disagreed on the set of top-3 models. * Practical Impact: In a highlighted dataset, traditional AUPR favored Logistic Regression, while AUPRG favored AdaBoost. The authors demonstrate that AdaBoost actually had a higher expected $F_1$ score, proving that AUPR can lead to suboptimal model selection.

Limitations and Future Work

  • Ranking Interpretation: While AUROC has a famous interpretation as the probability that a random positive is ranked higher than a random negative, a similar "probabilistic" interpretation for AUPRG is still being refined.
  • Non-uniform Weights: The current work assumes all positives are equally important. The authors suggest exploring how this relates to Discounted Cumulative Gain (DCG) for cases where some positives are more valuable than others.

Summary updated

Abstract

Precision-Recall analysis abounds in applications of binary classification where true negatives do not add value and hence should not affect assessment of the classifier's performance. Perhaps inspired by the many advantages of receiver operating characteristic (ROC) curves and the area under such curves for accuracy-based performance assessment, many researchers have taken to report Precision-Recall (PR) curves and associated areas as performance metric. We demonstrate in this paper that this practice is fraught with difficulties, mainly because of incoherent scale assumptions -- e.g., the area under a PR curve takes the arithmetic mean of precision values whereas the $F_{\beta}$ score applies the harmonic mean. We show how to fix this by plotting PR curves in a different coordinate system, and demonstrate that the new Precision-Recall-Gain curves inherit all key advantages of ROC curves. In particular, the area under Precision-Recall-Gain curves conveys an expected $F_1$ score on a harmonic scale, and the convex hull of a Precision-Recall-Gain curve allows us to calibrate the classifier's scores so as to determine, for each operating point on the convex hull, the interval of $\beta$ values for which the point optimises $F_{\beta}$. We demonstrate experimentally that the area under traditional PR curves can easily favour models with lower expected $F_1$ score than others, and so the use of Precision-Recall-Gain curves will result in better model selection.
BibTeX
@inproceedings{flach2015precision,
  author = {Peter A. Flach and Meelis Kull},
  title = {{Precision-Recall-Gain Curves: PR Analysis Done Right}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2015},
  url = {https://openreview.net/forum?id=r1V9B_ZubH},
}

Loading related content...

Figure