TL;DR
Traditional Precision-Recall (PR) curves are mathematically "broken" because they use an arithmetic scale to evaluate $F$-scores, which are based on harmonic means. This paper introduces Precision-Recall-Gain (PRG) curves, a new coordinate system that fixes these inconsistencies, allows for linear interpolation, and provides an Area Under the Curve (AUPRG) that actually relates to expected $F_1$ scores.
The Problem with Traditional PR Curves
In binary classification—especially in fields like information retrieval or medical diagnosis—we often deal with "imbalanced" data where negative examples far outnumber positives. In these cases, standard accuracy is a poor metric, and Receiver Operating Characteristic (ROC) curves can be overly optimistic because they account for True Negatives.
Researchers often turn to Precision-Recall (PR) curves instead. However, the authors argue that PR analysis is "fraught with difficulties." Specifically:
- Incoherent Scales: We evaluate models using the $F_1$ score (a harmonic mean), but PR curves use arithmetic interpolation between points.
- Non-Universal Baselines: The "random" baseline on a PR plot changes depending on the class distribution, making it hard to compare across datasets.
- Uninterpretable Area: The Area Under the PR Curve (AUPR) doesn't have a clear functional meaning relative to standard performance metrics.
Precision-Recall-Gain (PRG) Curves
The authors propose a change of coordinates from $(Recall, Precision)$ to $(Recall\ Gain, Precision\ Gain)$.
The "Gain" transformation rescales the metrics from the interval $[\pi, 1]$ (where $\pi$ is the proportion of positives) to $[0, 1]$ using a harmonic scale. The formulas are:
- Precision Gain ($precG$): $1 - \frac{\pi}{1-\pi} \cdot \frac{FP}{TP}$
- Recall Gain ($recG$): $1 - \frac{\pi}{1-\pi} \cdot \frac{FN}{TP}$
By moving to this "Gain" space, several mathematical properties align correctly:
- Linear Interpolation: A straight line between two points in PRG space represents a realizable weighted combination of those two classifiers. In traditional PR space, this interpolation is incorrectly assumed to be linear.
- $F_\beta$ Isometrics are Linear: Lines of constant $F_\beta$ score become straight lines with a slope of $-\beta^2$, whereas they are non-linear curves in standard PR space.
- Convex Hull Optimality: Just like in ROC analysis, you can construct a convex hull. Every point on this hull is optimal for some range of $\beta$ values, allowing for easy model calibration.
Key Insights
- Ignoring the "Bad" Zone: Any model with precision or recall lower than the class prior ($\pi$) is performing worse than an "always positive" classifier. PRG curves naturally ignore these regions by assigning them negative gain, focusing only on the "useful" part of the performance space.
- Meaningful Area: The Area Under the PRG Curve (AUPRG) is shown to be a linear version of the expected $F_1$ score. This gives the area a formal justification that AUPR lacks.
Empirical Validation
To prove that this isn't just a theoretical exercise, the authors conducted a massive empirical study using the OpenML platform, evaluating nearly 26,000 models across 886 binary classification tasks.
Key Results: * Model Selection Disagreement: In 24% of the tasks, AUPR and AUPRG disagreed on which model was the "best." * Top-3 Disagreement: In 58% of the tasks, the two metrics disagreed on the set of top-3 models. * Practical Impact: In a highlighted dataset, traditional AUPR favored Logistic Regression, while AUPRG favored AdaBoost. The authors demonstrate that AdaBoost actually had a higher expected $F_1$ score, proving that AUPR can lead to suboptimal model selection.
Limitations and Future Work
- Ranking Interpretation: While AUROC has a famous interpretation as the probability that a random positive is ranked higher than a random negative, a similar "probabilistic" interpretation for AUPRG is still being refined.
- Non-uniform Weights: The current work assumes all positives are equally important. The authors suggest exploring how this relates to Discounted Cumulative Gain (DCG) for cases where some positives are more valuable than others.
Summary updated