Compositional Training for End-to-End Deep AUC Maximization

Zhuoning Yuan, Zhishuai Guo, Nitesh Chawla, Tianbao YangICLR 2022 · OpenReview

TL;DR

Training deep neural networks to maximize the Area Under the ROC Curve (AUC) usually requires a clunky two-stage process because training from scratch leads to poor feature representations. This paper introduces Compositional DAM, a novel framework that embeds a "feature purification" step (using standard cross-entropy) inside an "AUC robustification" step, allowing for superior end-to-end training that outperforms traditional two-stage methods.

What Problem Does This Paper Address?

In many real-world scenarios, such as medical diagnosis or fraud detection, datasets are heavily imbalanced (e.g., many more healthy patients than sick ones). In these cases, standard accuracy is a poor metric, and researchers turn to Deep AUC Maximization (DAM).

However, DAM has a "dirty secret": you can't easily train it from scratch. If you try to maximize AUC directly on a raw network, the resulting feature representations are often "degraded" compared to those learned via standard Cross-Entropy (CE) loss. Consequently, the current state-of-the-art involves an ad-hoc two-stage approach: first, pretrain the network with CE loss, then fine-tune it with an AUC loss. This is cumbersome, requires manual tuning of when to switch stages, and doesn't necessarily yield the optimal unified model.

Methods

The authors propose a compositional objective function to solve this. Instead of choosing between CE and AUC, they nest them:

$$\min_{w} L_{AUC}(w - \alpha \nabla L_{AVG}(w))$$

Here, the "inner" function is a single gradient descent step aimed at minimizing the average loss (like CE), while the "outer" function is the AUC loss.

To optimize this non-standard objective, they developed the Primal-Dual Stochastic Compositional Adaptive (PDSCA) algorithm. Key features include:

  • Two-level updates: It alternatively takes a "feature purification" step (CE gradient) and a "classifier robustification" step (AUC gradient).
  • Moving Averages: It uses a moving average technique to track gradients, which is crucial for proving convergence in compositional optimization without requiring massive batch sizes.
  • Adaptive Learning Rates: It incorporates Adam-style adaptive step sizes to handle the complexities of deep learning optimization.

Datasets

The researchers evaluated their method on a wide array of imbalanced datasets:

  • Benchmarks: Binary versions of CatvsDog, CIFAR10, CIFAR100, and STL10 with varying imbalance ratios (1%, 10%, and 30%).
  • Medical Imaging: Naturally imbalanced datasets including Melanoma (skin cancer), CheXpert (chest X-rays), DDSM+ (mammograms), and PatchCam (histopathology).
  • Long-Tailed Benchmarks: CIFAR10-LT, CIFAR100-LT, STL10-LT, and ImageNet-LT for multi-class weighted loss experiments.

Key Insights and Ideas

  • The "Why" of Feature Degradation: AUC loss assigns different weights to positive and negative samples. While great for a classifier, this is bad for feature learning, where the network needs to see all examples roughly equally to learn low-level patterns (edges, textures, etc.).
  • 1 + 1 > 2: The authors show that their compositional approach is fundamentally different from—and better than—a simple linear combination ($L_{AUC} + cL_{AVG}$). A linear combination creates a trade-off where neither loss is fully optimized, whereas the compositional approach uses the CE gradient to "prepare" the weights for the AUC update.
  • Unified Training: By treating the training as a compositional problem, they eliminate the need for the "ad-hoc" decisions of two-stage training, such as which layers to freeze or exactly when to stop pretraining.

Key Results

  • Performance Boost: Compositional DAM (referred to as CT-AUC) consistently outperformed all baselines. On difficult medical imaging tasks, it improved the testing AUC score by 2% to 5% compared to training from scratch.
  • Better than Two-Stage: It yielded higher AUC scores than the standard two-stage fine-tuning approach, proving that end-to-end training is not just possible but preferable.
  • Faster Convergence: Empirical plots showed that the compositional method converges faster in terms of epochs compared to linear combinations or two-stage methods.
Training dynamics of the compositional objective components over epochs.
  • Visual Proof: t-SNE visualizations of the learned features showed that CT-AUC produces much cleaner clusters for positive and negative classes than AUC-from-scratch or CE-only methods.
t-SNE visualizations comparing feature representations learned by different methods.

Limitations and Future Work

  • Computational Overhead: The method requires calculating gradients of gradients (Hessian-vector products), which increases the per-iteration running time compared to simple CE. While the authors use a "first-order" trick to mitigate this, it remains more complex than standard SGD.
  • Hyperparameter Sensitivity: While the authors found good default values (e.g., $\beta \approx 0.9$), the framework introduces new hyperparameters like the inner step size $\alpha$.
  • Expansion: The authors suggest that this compositional framework could be extended to other non-decomposable loss functions beyond AUC, such as F-measure or Precision-Recall curves.

The code for this work is available as part of the LibAUC library.

Summary updated

Abstract

Recently, deep AUC maximization (DAM) has achieved great success in different domains (e.g., medical image classification). However, the end-to-end training for deep AUC maximization still remains a challenging problem. Previous studies employ an ad-hoc two-stage approach that first trains the network by optimizing a traditional loss (e.g., cross-entropy loss) and then finetunes the network by optimizing an AUC loss. This is because that training a deep neural network from scratch by maximizing an AUC loss usually does not yield a satisfactory performance. This phenomenon can be attributed to the degraded feature representations learned by maximizing the AUC loss from scratch. To address this issue, we propose a novel compositional training framework for end-to-end DAM, namely compositional DAM. The key idea of compositional training is to minimize a compositional objective function, where the outer function corresponds to an AUC loss and the inner function represents a gradient descent step for minimizing a traditional loss, e.g., the cross-entropy (CE) loss. To optimize the non-standard compositional objective, we propose an efficient and provable stochastic optimization algorithm. The proposed algorithm enhances the capabilities of both robust feature learning and robust classifier learning by alternatively taking a gradient descent step for the CE loss and for the AUC loss in a systematic way. We conduct extensive empirical studies on imbalanced benchmark and medical image datasets, which unanimously verify the effectiveness of the proposed method. Our results show that the compositional training approach dramatically improves both the feature representations and the testing AUC score compared with traditional deep learning approaches, and yields better performance than the two-stage approaches for DAM as well. The proposed method is implemented in our open-sourced library LibAUC (https://www.libauc.org) and code is available at https://github.com/Optimization-AI/LibAUC.
BibTeX
@inproceedings{yuan2022compositional,
  author = {Zhuoning Yuan and Zhishuai Guo and Nitesh Chawla and Tianbao Yang},
  title = {{Compositional Training for End-to-End Deep AUC Maximization}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=gPvB4pdu_Z},
}
Figure