Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic Choices

Dixian Zhu, Xiaodong Wu, Tianbao Yang

Introduction

In domains such as medical diagnosis and drug discovery, datasets are frequently characterized by extreme class imbalance. In these scenarios, standard accuracy is a misleading metric; a model can achieve high accuracy by simply predicting the majority class. The Area Under the ROC Curve (AUROC) is the gold standard for evaluation because it measures the model's ability to rank positive samples higher than negative ones. However, directly optimizing for AUROC is non-trivial due to its non-differentiable nature. This study provides a comprehensive benchmark of deep AUROC maximization, evaluating 10 loss functions and various algorithmic choices across 14,400 unique experimental runs.

Methodology: The Three Pillars of AUROC Optimization

The researchers investigated three critical components of the optimization pipeline: loss functions, optimizers, and algorithmic "knobs" such as sampling rates and normalization.

1. Loss Functions: Pairwise vs. Composite

The benchmark categorizes loss functions into two main families:

  • Pairwise Losses: These traditional methods compare every positive sample to every negative sample within a batch. While effective, they are computationally expensive as the number of pairs grows quadratically with batch size.
  • Composite Losses: A newer class of functions that reformulate the AUROC objective using class-wise statistics (means and variances). This approach avoids the overhead of explicit pair generation, making them significantly faster for deep learning applications.
Formulations of common pairwise loss functions used for AUROC optimization.
Formulations of composite loss functions, which utilize class-wise statistics to improve efficiency.

The study found that composite losses, particularly the Composite Squared Hinge (CSH), often outperform pairwise losses in both speed and generalization across diverse datasets.

2. Optimizers and the "Adam Trap"

A significant finding of this benchmark is the performance gap between Adam and Momentum-style optimizers. While Adam-type optimizers are highly efficient at reducing training loss and converge quickly, they frequently fail to generalize in AUROC optimization tasks. On image datasets like CIFAR10, Momentum-style methods consistently achieved higher test AUROC scores (e.g., ~0.911) compared to Adam (~0.868). For practitioners, Momentum remains the safer choice for final model performance.

3. Algorithmic Best Practices

Beyond the choice of loss and optimizer, several algorithmic "knobs" significantly impact performance:

  • Sampling Positive Rate (SPR): Increasing the proportion of positive samples in each mini-batch—often up to 50%—is essential for boosting AUROC in highly imbalanced settings.
  • Output Normalization: Applying Sigmoid activation or $\ell_2$ score normalization to the final layer prevents extreme outlier predictions and consistently improves results across different loss functions.
  • Noise Resilience: In datasets with corrupted or "noisy" labels (like the large-scale CheXpert chest X-ray dataset), symmetric losses like the Pairwise Barrier Hinge (PBH) show superior robustness.

Experimental Setup and Datasets

The benchmark utilized 12 datasets spanning two distinct domains to ensure the generalizability of the findings:

  • Computer Vision: STL10, CIFAR10, CIFAR100, and medical imaging datasets like Melanoma and the large-scale CheXpert.
  • Molecular Graphs: HIV, MUV, Tox21, and ToxCast, which are naturally imbalanced and used for predicting chemical properties.

Conclusion

This large-scale study clarifies the best practices for deep AUROC optimization. The results suggest a shift toward composite loss functions and Momentum-based optimizers, combined with strategic over-sampling and output normalization. These findings provide a robust roadmap for researchers and practitioners dealing with imbalanced data in high-stakes applications, moving beyond traditional pairwise methods toward more efficient and generalizable composite approaches.

Summary updated

Abstract

The area under the ROC curve (AUROC) has been vigorously applied for imbalanced classification and moreover combined with deep learning techniques. However, there is no existing work that provides sound information for peers to choose appropriate deep AUROC maximization techniques. In this work, we fill this gap from three aspects. (i) We benchmark a variety of loss functions with different algorithmic choices for deep AUROC optimization problem. We study the loss functions in two categories: pairwise loss and composite loss, which includes a total of 10 loss functions. Interestingly, we find composite loss, as an innovative loss function class, shows more competitive performance than pairwise loss from both training convergence and testing generalization perspectives. Nevertheless, data with more corrupted labels favors a pairwise symmetric loss. (ii) Moreover, we benchmark and highlight the essential algorithmic choices such as positive sampling rate, regularization, normalization/activation, and optimizers. Key findings include: higher positive sampling rate is likely to be beneficial for deep AUROC maximization; different datasets favors different weights of regularizations; appropriate normalization techniques, such as sigmoid and $\ell_2$ score normalization, could improve model performance. (iii) For optimization aspect, we benchmark SGD-type, Momentum-type, and Adam-type optimizers for both pairwise and composite loss. Our findings show that although Adam-type method is more competitive from training perspective, but it does not outperform others from testing perspective.
BibTeX
@misc{zhu2022benchmarking,
  author = {Dixian Zhu and Xiaodong Wu and Tianbao Yang},
  title = {{Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic Choices}},
  year = {2022},
  eprint = {2203.14177},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.14177},
}
Figure