Abstract
Generalized Linear Models (GLMs) are foundational in machine learning but highly susceptible to adversarial label corruption. We introduce SVAM (Sequential Variance-Altered MLE), a unified framework that achieves robustness by iteratively adjusting the variance of the likelihood function. Starting with a "blurry" likelihood to mitigate outlier influence and gradually "sharpening" focus, SVAM provides strong theoretical guarantees and superior empirical performance. Our results demonstrate that SVAM handles significant corruption fractions and converges reliably regardless of initialization.
Problem Statement: The Vulnerability of GLMs
Generalized Linear Models (GLMs), including logistic and gamma regression, are workhorses of statistical modeling. However, they are notoriously sensitive to adversarial label corruption. Even a small fraction of maliciously modified training labels—such as flipped binary labels or extreme outliers in regression—can cause standard Maximum Likelihood Estimation (MLE) to fail completely.
Existing robust methods often suffer from three main drawbacks:
- Lack of Generality: Many are tailored to specific loss functions (e.g., squared error) and do not generalize across the exponential family.
- Susceptibility to Adaptive Adversaries: They may fail when an attacker strategically chooses corruptions based on the dataset or model.
- Weak Theoretical Guarantees: Many algorithms lack a proven "breakdown point"—the maximum fraction of corruption an algorithm can tolerate before its error becomes unbounded.
The SVAM Framework: Likelihood Annealing
The core innovation of this work is SVAM (Sequential Variance-Altered MLE). Unlike traditional weighted MLE approaches, SVAM employs a novel variance-reduction technique that functions similarly to simulated annealing.
1. Variance Altering
The framework utilizes a transformation that scales the variance of an exponential family distribution while preserving its mode and the relative ordering of likelihoods.
2. The "Blurry" Start and Sequential Sharpening
The algorithm begins with a very high variance (low $\beta$). In this initial "blurry" state, the likelihood function is relatively flat, ensuring that all data points—clean or corrupted—receive comparable weights. This prevents the model from being immediately derailed by adversarial outliers. As iterations proceed, the algorithm gradually increases $\beta$ (decreases variance). As the model estimate improves, the likelihood of uncorrupted points naturally dominates, allowing the algorithm to "home in" on the clean data.
3. Universal Applicability
Because SVAM operates on the underlying distribution's variance, it can be applied to any GLM by simply swapping the distribution (e.g., Gaussian for least squares, Gamma for positive-valued regression).
Key Results and Discussion
The authors evaluated SVAM against several state-of-the-art robust methods (e.g., TORRENT, SEVER, STIR) using synthetic benchmarks designed to test the limits of adversarial robustness.
Robustness to Corruption Levels
A critical metric for robust algorithms is the breakdown point. As shown in the following analysis, SVAM-RR maintains a near-zero error even as the corruption fraction $\alpha$ increases towards 0.4, significantly outperforming methods like SEVER and STIR which break down much earlier.
Global Convergence and Initialization
Many robust algorithms require a "warm start" or a good initial guess to avoid local minima created by outliers. SVAM, however, exhibits remarkable robustness to initialization. Whether initialized at the origin or with adversarially corrupted data, the algorithm consistently converges to the true model parameters.
Performance Highlights
- Speed: In least-squares tasks, SVAM-RR is often twice as fast as specialized competitors like TORRENT.
- Exact Recovery: In the absence of random noise, SVAM can recover exact true parameters even against a fully adaptive adversary.
- Gamma Regression: SVAM provides the first provable guarantees for robust Gamma regression, a task previously lacking a known breakdown point.
Limitations and Future Work
While SVAM represents a significant step forward, some challenges remain. The algorithm's performance is somewhat sensitive to hyperparameters like the initial scale ($\beta_1$) and the increment factor ($\xi$). Furthermore, while SVAM provides a breakdown point for Gamma regression, it currently scales with $1/\sqrt{d}$ (where $d$ is dimensionality), suggesting potential fundamental limits compared to the constant breakdown point achieved in least-squares. Future research will explore how these methods perform under "dirty" feature distributions in real-world datasets, moving beyond label-only corruption.
Summary updated