Introduction: The Class Imbalance Trap
In many critical classification domains, such as Intrusion Detection Systems (IDS) or medical diagnostics, the data is heavily imbalanced. A typical IDS might process $10^5$ normal events for every single attack. In these scenarios, standard evaluation metrics can be dangerously misleading. This paper introduces Bayesian-ROC (B-ROC) curves, a visualization tool designed to provide a more intuitive and practical assessment of classifier performance when the minority class is extremely rare.
The Base-Rate Fallacy in ROC Analysis
The primary issue with standard Receiver Operating Characteristic (ROC) curves, which plot the Probability of Detection ($P_D$) against the False Alarm Rate ($P_{FA}$), is their independence from class distribution. This leads to the base-rate fallacy.
For example, a classifier with a 1% false alarm rate ($P_{FA} = 0.01$) and perfect detection ($P_D = 1$) sounds excellent. However, with a 1:100,000 imbalance, that 1% false alarm rate results in 1,000 false alarms for every single true attack detected. On a standard ROC curve, the performance appears near-perfect because the "area of interest" is compressed into a tiny sliver near the origin.
Introducing Bayesian-ROC (B-ROC) Curves
To address this, the authors propose the B-ROC curve. The fundamental change is the x-axis: instead of $P_{FA}$, the B-ROC plots the Bayesian False Alarm Rate ($B_{FA}$).
$B_{FA}$ is defined as $1 - PPV$ (Positive Predictive Value), representing the probability that a given alarm is actually a false alarm ($Pr[C=0 | A=1]$). This metric offers immediate operational intuition: it tells a human operator the likelihood that an alarm is a waste of time.
The mapping from a standard ROC point to a B-ROC point is given by: $$B_{FA} = \frac{(1-p)P_{FA}}{pP_D + (1-p)P_{FA}}$$ where $p$ is the prior probability of the minority class (the "attack").
Sensitivity to Prior Probabilities
A key advantage of the B-ROC framework is its ability to handle uncertainty regarding the class distribution. Because $B_{FA}$ depends on $p$, researchers can plot a family of B-ROC curves for different values of $p$ (e.g., from $10^{-1}$ down to $10^{-7}$).
This visualization reveals how a classifier's utility degrades as the data becomes more imbalanced. A model that appears superior at one level of imbalance might become inferior at another, a nuance that standard ROC curves completely obscure.
Key Insights and Operational Utility
B-ROC curves effectively "zoom in" and stretch the critical region near the origin of an ROC curve across the entire x-axis. This makes performance differences visible and allows for cost-free comparison. Unlike decision-theoretic approaches that require assigning specific costs to false positives and negatives—which are often unknown—B-ROC curves allow for visual comparison across a range of imbalance severities.
Conclusion
Standard ROC curves fail to provide the necessary intuition for choosing operational points in heavily imbalanced scenarios. By shifting the focus to the Bayesian false alarm rate, B-ROC curves provide a more realistic picture of real-world performance, helping practitioners avoid the base-rate fallacy and select classifiers that remain effective even when the target class is exceptionally rare.
Summary updated