Abstract
This paper introduces a simple yet powerful algorithm for minimizing a "mean-plus-standard-deviation" objective, even when data is messy or heavy-tailed. By jointly learning the model parameters alongside a robust location and scale, the proposed "Modified Sun-Huber" method outperforms traditional techniques like CVaR and DRO without requiring the manual "hyperparameter gymnastics" usually needed to handle outliers.
Introduction
In traditional machine learning, the standard goal is to minimize the average loss. However, average performance can be misleading, especially when a model performs well on the majority of data but fails catastrophically on a small subset. To address this, researchers often employ mean-variance or mean-standard deviation (mean-SD) objectives, which penalize both the average error and its volatility.
While theoretically sound, these objectives are notoriously difficult to optimize due to non-convexity. Existing robust alternatives, such as Distributionally Robust Optimization (DRO) and Conditional Value-at-Risk (CVaR), are convex but highly sensitive to outliers and heavy-tailed noise. In the presence of "wild" data points, these methods can overfit to outliers, leading to poor generalization.
Methods
The proposed solution is the Modified Sun-Huber objective, which introduces "concomitant scaling." Instead of pre-estimating variance, the algorithm simultaneously learns model weights ($h$), a robust center ($a$), and a scale parameter ($b$) for the loss distribution.
The core of this approach is a "pseudo-Huber" loss function $\rho$, which provides a smooth robust alternative to standard loss functions.
The key innovation lies in the transformation $x \mapsto b^2 \rho(x/b)$. By treating the scale $b$ as a trainable parameter, the model can adaptively robustify the objective based on the observed volatility of the loss.
The paper provides a principled heuristic for setting internal parameters based on sample size $n$: $\beta \propto 1/\sqrt{n}$ and $\alpha = \beta$. This turns a complex theoretical framework into a "plug-and-play" routine compatible with standard stochastic gradient descent (SGD).
Experimental Results
The method was evaluated on synthetic 2D toy problems and real-world benchmarks including adult, australian, cifar10, and fashion_mnist.
- Outlier Resilience: In 2D simulations, while Vanilla ERM, CVaR, and DRO decision boundaries were significantly skewed by a single distant outlier, the Modified Sun-Huber method remained stable.
- Performance: On real-world datasets, the proposed method consistently achieved lower (better) mean-SD values on test data compared to benchmarks.
- Simplicity: A major advantage is that the method worked effectively across different datasets using the same parameter logic, whereas CVaR and DRO required extensive tuning of robustness levels.
Discussion and Limitations
The work successfully bridges robust mean estimation and risk minimization. It demonstrates that a "good-enough" scale parameter is sufficient to provide sub-Gaussian-like performance guarantees. Although the objective is non-convex and non-smooth, the resulting stability in the presence of heavy tails justifies this trade-off.
However, optimization complexity remains a challenge. For very complex tasks, standard SGD might require more fine-tuned settings of $\alpha$ and $\beta$. Future research directions include developing "optimizer-aligned" versions of this objective and providing detailed theoretical analysis of generalization error for specific implementations.
Conclusion
The Modified Sun-Huber method offers a robust and practical alternative for variance-regularized risk minimization. By incorporating concomitant scaling, it achieves superior resilience to outliers and heavy-tailed noise with minimal hyperparameter tuning.
Summary updated