Making Robust Generalizers Less Rigid with Soft Ascent-Descent

Matthew J. Holland, Toma HamadaarXiv 2024 · DOI · arXiv

Abstract

Existing robust machine learning methods like SharpDRO often fail when applied to smaller, less expressive models where the gap between "easy" and "hard" data points is wide. This paper introduces Concentrated OCE (COCE), a state-agnostic training objective that penalizes the dispersion of transformed losses. COCE offers a low-cost, effective alternative to sharpness-aware minimization that excels even when the model isn't overparameterized.

Introduction

In machine learning, while average performance is the standard metric, high-stakes applications demand models that perform reliably on "hard" or rare data points, such as corrupted images or under-represented groups. This challenge is the core of robust generalization. Current state-of-the-art methods, such as Sharpness-Aware Minimization (SAM) and SharpDRO, attempt to minimize "sharpness"—the sensitivity to weight perturbations—specifically on worst-case examples.

However, these methods often rely on overparameterization. In large neural networks, the performance gap between easy and hard points eventually narrows. When moving to simpler, constrained models, these methods often "break down" as the difficulty gap becomes extreme.

The Problem: Rigidity in Robustness

Robustness techniques designed for infinite-capacity models do not necessarily translate to constrained architectures. For instance, in a basic CNN, SharpDRO's performance can collapse as it over-focuses on the hardest points at the expense of general performance. Furthermore, many existing methods are "state-aware," requiring knowledge of specific data conditions (e.g., corruption levels) during training.

Examples of image corruptions at various severity levels (0-5) used to test robust generalization.

Methodology: Concentrated OCE (COCE)

The authors propose Concentrated OCE (COCE), a state-agnostic approach that identifies difficult points implicitly by focusing on the concentration of the loss distribution. The method involves two primary steps:

  1. Loss Transformation: Instead of raw loss, a convex, non-decreasing transformation $\phi(\cdot)$ (such as exponential tilting or Conditional Value-at-Risk) is applied. This transformation shifts the model's focus toward the "tail" of the loss distribution—the large, infrequent losses associated with hard examples.
  2. Penalizing Dispersion: COCE penalizes the dispersion of these transformed losses around a fixed threshold $\eta$. By minimizing this dispersion, the model is forced to produce more consistent performance across the dataset.
Examples of valid transformation functions $\phi(\cdot)$ and dispersion penalty functions $\rho(\cdot)$ used in the COCE framework.

A key insight is that when the loss crosses the threshold $\eta$, the gradient update naturally switches between "ascent" and "descent." This creates a "soft" version of SAM. Crucially, while SAM requires two gradient computations per update, COCE achieves a similar regularizing effect with only one, significantly improving computational efficiency.

Experimental Results

The researchers evaluated COCE on CIFAR-10, CIFAR-100, and ImageNet-30 using corrupted versions of these datasets. They compared a large Wide ResNet (WRN-28-2) against a simpler CNN.

Key Findings:

  • Superior Balanced Accuracy: On the small CNN model, COCE significantly outperformed SharpDRO. While SharpDRO’s accuracy dropped sharply during hyperparameter tuning toward the worst-case, COCE remained robust and stable.
  • Efficiency: COCE provided performance comparable to or better than SAM but at half the gradient computation cost.
  • Implicit Regularization: COCE leads to significantly smaller model norms (L2 norm of weights) compared to standard Empirical Risk Minimization (ERM) or SAM, preventing the model from becoming too "rigid."
  • Model Scale Dynamics: While SharpDRO remains strong for very large models, COCE is the clear winner for smaller, more constrained models, filling a critical gap in robustness literature.

Conclusion and Future Work

COCE demonstrates that state-agnosticism and efficiency are achievable in robust generalization. By focusing on loss concentration rather than explicit sharpness, it provides a versatile objective for various model scales. Future work will focus on addressing hyperparameter sensitivity—specifically the selection of $\phi$ and $\eta$—and extending the "concentrated" approach to domains like algorithmic fairness and class imbalance.

Summary updated

Abstract

While the traditional formulation of machine learning tasks is in terms of performance on average, in practice we are often interested in how well a trained model performs on rare or difficult data points at test time. To achieve more robust and balanced generalization, methods applying sharpness-aware minimization to a subset of worst-case examples have proven successful for image classification tasks, but only using overparameterized neural networks under which the relative difference between "easy" and "hard" data points becomes negligible. In this work, we show how such a strategy can dramatically break down under simpler models where the difficulty gap becomes more extreme. As a more flexible alternative, instead of typical sharpness, we propose and evaluate a training criterion which penalizes poor loss concentration, which can be easily combined with loss transformations such exponential tilting, conditional value-at-risk (CVaR), or distributionally robust optimization (DRO) that control tail emphasis.
BibTeX
@misc{holland2024making,
  author = {Matthew J. Holland and Toma Hamada},
  title = {{Making Robust Generalizers Less Rigid with Soft Ascent-Descent}},
  year = {2024},
  eprint = {2408.03619},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.03619},
}
Figure