Multiclass ROC

Liang Wang, Luis CarvalhoarXiv 2024 · DOI · arXiv

Abstract

Evaluating multiclass machine learning models often lacks the intuitive visual clarity and robustness of binary Receiver Operating Characteristic (ROC) analysis. We present ROC-DMF, a framework based on Deviance Matrix Factorization that generalizes ROC curves to multiclass settings. By decomposing the problem into pairwise class interactions and applying rank-one matrix factorization, ROC-DMF produces interpretable 2D plots and Area Under the Curve (AUC) scores that are invariant to class imbalance and support cost-sensitive evaluation.

Introduction

In binary classification, the ROC curve and its associated AUC are gold-standard tools for summarizing model performance across all decision thresholds. However, extending these concepts to multiclass problems (e.g., identifying species or digits) is non-trivial. Existing generalizations often fail to provide human-interpretable visualizations, are overly sensitive to class imbalance, or cannot easily incorporate varying misclassification costs.

The ROC-DMF Framework

The proposed ROC-DMF (Deviance Matrix Factorization) framework addresses these challenges by treating multiclass evaluation as a dimensionality reduction problem.

Pairwise Decomposition

The core idea is to break down the $k$-class problem into $k(k-1)$ pairwise comparisons. For each class $i$, the model's ability to distinguish it from every other class $j$ is evaluated.

The ROC-DMF framework decomposes the multiclass problem into $k(k-1)$ pairwise class interactions, represented here as conditional subsets.

This process generates two large matrices: $M^{tp}$ for True Positive Rates and $M^{fp}$ for False Positive Rates. These matrices capture the complex web of interactions between all class pairs across multiple thresholds.

Matrix Factorization

To distill this complexity into a single ROC curve, the authors apply Binomial Matrix Factorization. By utilizing a rank-one factorization ($q=1$), the high-dimensional pairwise performance data is compressed into a one-dimensional latent vector. This vector is then mapped back into a 2D space, creating a "multiclass equivalent" ROC plot that retains the intuitive properties of its binary counterpart.

Furthermore, a weighting parameter $Q_{ij}$ allows users to assign specific importance to different class pairs, enabling the direct integration of misclassification costs into the metric.

Experimental Validation

The authors validated ROC-DMF using both simulated data and real-world benchmarks from the UCI Machine Learning Repository, including the Car, Glass, Iris, Optical, Pen, and Tae datasets.

Key Findings

  • Robustness to Skewness: Unlike standard accuracy, ROC-DMF remains stable even when class distributions are highly imbalanced. Because it operates on rates (TPR/FPR) rather than raw counts, the majority class does not disproportionately influence the final curve.
  • Cost Sensitivity: Experiments demonstrated that adjusting the cost parameter $c$ predictably shifts the ROC curve, allowing practitioners to prioritize specific types of errors (e.g., avoiding false negatives in medical diagnosis).
  • Visualizing Uncertainty: By employing bootstrapping techniques within the binomial model, the framework can generate "clouds" of ROC curves. This provides a visual representation of the model's performance variance, which is critical for high-stakes decision-making.

Conclusion and Future Work

ROC-DMF successfully bridges the gap between binary and multiclass evaluation, offering a visual, robust, and cost-sensitive alternative to traditional metrics. While the rank-one approach provides excellent interpretability, future research may explore higher-rank factorizations to capture more intricate class relationships, as well as optimizing the quadratic computational scaling for datasets with very large numbers of classes.

Summary updated

BibTeX
@misc{wang2024multiclass,
  author = {Liang Wang and Luis Carvalho},
  title = {{Multiclass ROC}},
  year = {2024},
  eprint = {2404.13147},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.13147},
}
Figure