Robust variance-regularized risk minimization with concomitant scaling

Matthew J. Holland

Abstract

Under losses which are potentially heavy-tailed, we consider the task of minimizing sums of the loss mean and standard deviation, without trying to accurately estimate the variance. By modifying a technique for variance-free robust mean estimation to fit our problem setting, we derive a simple learning procedure which can be easily combined with standard gradient-based solvers to be used in traditional machine learning workflows. Empirically, we verify that our proposed approach, despite its simplicity, performs as well or better than even the best-performing candidates derived from alternative criteria such as CVaR or DRO risks on a variety of datasets.

Introduction

Standard machine learning usually aims for Empirical Risk Minimization (ERM)—minimizing the average loss. While this works well for "nice" data, it fails when data is heavy-tailed or contains outliers, as the average is easily pulled by extreme values. A more robust goal is the mean-SD criterion: minimizing the sum of the mean loss and its standard deviation (SD). This penalizes models that have high variability in their performance. However, accurately estimating the standard deviation under heavy tails is notoriously difficult. Existing alternatives like Distributionally Robust Optimization (DRO) or Conditional Value-at-Risk (CVaR) can be computationally heavy or require sensitive hyperparameter tuning. This work introduces a "plug-and-play" objective that is robust, theoretically grounded, and easy to use with standard gradient-based optimizers.

The Modified Sun-Huber Objective

The core contribution is the modified Sun-Huber objective. Instead of a standard squared loss, it uses a "pseudo-Huber" function $\rho(x) = \sqrt{x^2 + 1} - 1$, which acts like a quadratic function near zero but grows linearly for large values, naturally dampening the influence of outliers.

The objective function $C_n(h; a, b)$ involves three components optimized simultaneously: the model parameters $h$, a location parameter $a$ (acting as a robust estimate of the mean), and a scale parameter $b$. This joint optimization, termed "concomitant scaling," allows the algorithm to automatically adjust its sensitivity to the scale of the losses. The paper provides a principled way to set internal hyperparameters based solely on the sample size $n$, removing the need for manual "magic number" tuning.

Empirical Evaluation

The method was evaluated on both synthetic and real-world benchmarks, including linear classification and image classification tasks (CIFAR-10, Fashion-MNIST).

Outlier Resistance

In a 2D simulation, the robustness of the Modified Sun-Huber method is clearly demonstrated. While Vanilla ERM is often derailed by extreme values, the proposed method effectively ignores noise to find the correct decision boundary.

Figure 4: 2D linear classification simulation showing the Modified Sun-Huber method's resistance to a distant outlier (marked 'x').

Stability and Performance

On real-world datasets, the proposed method achieved significantly lower mean-SD values on test data compared to ERM, CVaR, and $\chi^2$-DRO. Notably, the trajectories of the objective over training epochs were much smoother and reached lower final error rates than competing robust methods, which often showed high sensitivity to their own robustness hyperparameters.

Figure 6: Comparison of training trajectories for error rate and norm across different robust optimization methods.

Discussion and Future Work

While the method works well in practice with SGD, the joint objective is technically non-convex and non-smooth. Empirical tests show that standard stochastic gradient descent handles it remarkably well, often outperforming theoretically "nicer" convex alternatives. Future work may involve designing optimizers specifically tailored to the geometry of this new objective function to guarantee faster or more stable convergence, and further exploring the "Bias-Robustness" trade-off inherent in the $\alpha$ and $\beta$ parameters.

Summary updated

Abstract

Under losses which are potentially heavy-tailed, we consider the task of minimizing sums of the loss mean and standard deviation, without trying to accurately estimate the variance. By modifying a technique for variance-free robust mean estimation to fit our problem setting, we derive a simple learning procedure which can be easily combined with standard gradient-based solvers to be used in traditional machine learning workflows. Empirically, we verify that our proposed approach, despite its simplicity, performs as well or better than even the best-performing candidates derived from alternative criteria such as CVaR or DRO risks on a variety of datasets.
BibTeX
@inproceedings{holland2024robust,
  author = {Matthew J. Holland},
  title = {{Robust variance-regularized risk minimization with concomitant scaling}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=hHQxpG2AC3},
}
Figure