B-ROC Curves for the Assessment of Classifiers over Imbalanced Data Sets

Alvaro A. Cárdenas, John S. BarasAAAI 2006

Introduction: The Class Imbalance Trap

In many critical classification domains, such as Intrusion Detection Systems (IDS) or medical diagnostics, the data is heavily imbalanced. A typical IDS might process $10^5$ normal events for every single attack. In these scenarios, standard evaluation metrics can be dangerously misleading. This paper introduces Bayesian-ROC (B-ROC) curves, a visualization tool designed to provide a more intuitive and practical assessment of classifier performance when the minority class is extremely rare.

The Base-Rate Fallacy in ROC Analysis

The primary issue with standard Receiver Operating Characteristic (ROC) curves, which plot the Probability of Detection ($P_D$) against the False Alarm Rate ($P_{FA}$), is their independence from class distribution. This leads to the base-rate fallacy.

For example, a classifier with a 1% false alarm rate ($P_{FA} = 0.01$) and perfect detection ($P_D = 1$) sounds excellent. However, with a 1:100,000 imbalance, that 1% false alarm rate results in 1,000 false alarms for every single true attack detected. On a standard ROC curve, the performance appears near-perfect because the "area of interest" is compressed into a tiny sliver near the origin.

Zoom of the Area of Interest in a standard ROC curve, showing the tiny region relevant for imbalanced datasets.

Introducing Bayesian-ROC (B-ROC) Curves

To address this, the authors propose the B-ROC curve. The fundamental change is the x-axis: instead of $P_{FA}$, the B-ROC plots the Bayesian False Alarm Rate ($B_{FA}$).

$B_{FA}$ is defined as $1 - PPV$ (Positive Predictive Value), representing the probability that a given alarm is actually a false alarm ($Pr[C=0 | A=1]$). This metric offers immediate operational intuition: it tells a human operator the likelihood that an alarm is a waste of time.

The mapping from a standard ROC point to a B-ROC point is given by: $$B_{FA} = \frac{(1-p)P_{FA}}{pP_D + (1-p)P_{FA}}$$ where $p$ is the prior probability of the minority class (the "attack").

Sensitivity to Prior Probabilities

A key advantage of the B-ROC framework is its ability to handle uncertainty regarding the class distribution. Because $B_{FA}$ depends on $p$, researchers can plot a family of B-ROC curves for different values of $p$ (e.g., from $10^{-1}$ down to $10^{-7}$).

A family of B-ROC curves plotted for varying prior probabilities $p$.

This visualization reveals how a classifier's utility degrades as the data becomes more imbalanced. A model that appears superior at one level of imbalance might become inferior at another, a nuance that standard ROC curves completely obscure.

Key Insights and Operational Utility

B-ROC curves effectively "zoom in" and stretch the critical region near the origin of an ROC curve across the entire x-axis. This makes performance differences visible and allows for cost-free comparison. Unlike decision-theoretic approaches that require assigning specific costs to false positives and negatives—which are often unknown—B-ROC curves allow for visual comparison across a range of imbalance severities.

Conclusion

Standard ROC curves fail to provide the necessary intuition for choosing operational points in heavily imbalanced scenarios. By shifting the focus to the Bayesian false alarm rate, B-ROC curves provide a more realistic picture of real-world performance, helping practitioners avoid the base-rate fallacy and select classifiers that remain effective even when the target class is exceptionally rare.

Summary updated

BibTeX
@inproceedings{cardenas2006b,
  author = {Alvaro A. Cárdenas and John S. Baras},
  title = {{B-ROC Curves for the Assessment of Classifiers over Imbalanced Data Sets}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure