TL;DR
This paper empirically analyzes nine common supervised learning performance metrics, revealing that they occupy distinct "metric spaces" and that no single metric is universally optimal. The authors introduce a new composite metric, SAR, that aims to provide a robust, general-purpose evaluation tool by combining accuracy, ROC area, and squared error.
What Problem Does This Paper Address?
Evaluating supervised learning models is crucial, but choosing the right performance metric is challenging. Different metrics capture different aspects of model performance, and a model that excels on one metric might perform poorly on another. This makes it difficult to select the best model, especially when the "true" underlying model is unknown and we must rely on empirical evaluation. The paper aims to shed light on the relationships between common performance metrics and to identify a metric that offers a good balance across different evaluation criteria.
Methods
The researchers trained a large number of models (2000 per dataset, across 7 datasets) using seven different learning algorithms: Neural Networks (ANN), Support Vector Machines (SVM), Bagged Decision Trees (BAG-DT), Boosted Decision Trees (BST-DT), Boosted Decision Stumps (BST-STMP), single Decision Trees (DT), and k-Nearest Neighbors (KNN). For each model, they evaluated performance across nine metrics: Accuracy (ACC), Lift (LFT), F-Score (FSC), Area under the ROC Curve (AUC), Average Precision (APR), Precision/Recall Break-Even Point (BEP), Root Mean Squared Error (RMS), Mean Cross Entropy (MXE), and Probability Calibration (CAL).
To understand the relationships between these metrics, they employed two primary techniques:
- Multidimensional Scaling (MDS): This technique visualizes the "distance" between metrics in a lower-dimensional space. Metrics that are similar in their evaluation behavior will cluster together. They performed MDS using both normalized scores and standard deviation scaling to account for the unique behavior of the CAL metric.
- Correlation Analysis: They calculated rank correlations between pairs of metrics to quantify how similarly they rank models.
A novel metric, SAR (Squared error, Accuracy, ROC area), was proposed and evaluated. SAR is defined as (ACC + AUC + (1 - RMS)) / 3.
Datasets
The study used seven binary classification datasets from the UCI Repository and other sources:
- ADULT
- COVER-TYPE
- LETTER (two versions: unbalanced LETTER.P1 and balanced LETTER.P2)
- HYPER_SPECT
- SLAC
- MEDIS
Key Insights and Ideas
- Metric Spaces: The nine performance metrics occupy distinct "metric spaces." MDS plots reveal that metrics can be broadly grouped: probability metrics (RMS, MXE, CAL) tend to be separate from ordering/rank metrics (AUC, APR, BEP, LFT), with threshold metrics (ACC, FSC) falling in between.
- CAL is an Outlier: Probability Calibration (CAL) is consistently distant from most other metrics, suggesting it captures a unique aspect of model performance that is not well-represented by other common metrics.
- Ordering Metrics are Similar: AUC, APR, and BEP are highly correlated and cluster together, indicating they are largely interchangeable for evaluating model ranking capabilities. Lift also shows high correlation with these ordering metrics.
- Accuracy's Unique Position: Accuracy (ACC) doesn't correlate as strongly with other threshold metrics (like FSC) as one might expect, and its relationship with ordering metrics is also somewhat surprising.
- Maximum Margin Methods Trade-offs: Boosting and SVMs excel on metrics like accuracy and ROC area but perform poorly on probability metrics (RMS, MXE, CAL). This is attributed to their "maximum margin" nature, which prioritizes clear separation boundaries over precise probability estimation. Platt scaling helps SVMs mitigate this issue for probability metrics.
- Squared Error is Robust: RMS is surprisingly centrally located in the MDS plots and shows high average correlation with other metrics, suggesting it's a good general-purpose metric.
- SAR as a Generalist: The proposed SAR metric shows promise as a general-purpose metric, demonstrating good correlation with others and a central location in MDS plots, though its advantage over RMS alone is modest.
Key Results
- High Correlations: Several pairs of metrics exhibit very high rank correlations (above 0.90), including LFT/AUC (0.96), AUC/APR (0.95), and ACC/BEP (0.93).
- Low Correlation for CAL: CAL has the lowest average rank correlation (0.63) with other metrics, highlighting its distinctiveness.
- Learning Algorithm Performance: Neural Networks (ANNs) and SVMs (with Platt scaling) show excellent overall performance across most metrics. Boosted trees excel on threshold and ordering metrics but struggle with probability metrics.
- SAR's Performance: SAR outperformed eight of the nine individual metrics in model selection experiments, tying with RMS.
Limitations and Future Work
- CAL Normalization: The paper notes challenges in normalizing CAL scores due to the baseline model having good calibration, which affects its placement in MDS plots.
- SAR Improvement: While SAR is a good general-purpose metric, the authors suggest further work to devise a metric that yields more improvement over RMS alone.
- Metric Space Dimensions: The metric space appears to have 3-5 significant dimensions, indicating that no single metric can fully capture all aspects of performance.
- Future Research: The authors hope for further research to design better metrics and to better understand which metrics are most appropriate for different settings. They also suggest investigating other measures of calibration.
Summary updated