Abstract
For many inference problems in statistics and econometrics, the unknown parameter is identified by a set of moment conditions. A generic method of solving moment conditions is the Generalized Method of Moments (GMM). However, classical GMM estimation is potentially very sensitive to outliers. Robustified GMM estimators have been developed in the past, but suffer from several drawbacks: computational intractability, poor dimension-dependence, and no quantitative recovery guarantees in the presence of a constant fraction of outliers. In this work, we develop the first computationally efficient GMM estimator (under intuitive assumptions) that can tolerate a constant $\epsilon$ fraction of adversarially corrupted samples, and that has an $\ell_2$ recovery guarantee of $O(\sqrt{\epsilon})$. To achieve this, we draw upon and extend a recent line of work on algorithmic robust statistics for related but simpler problems such as mean estimation, linear regression and stochastic optimization. As two examples of the generality of our algorithm, we show how our estimation algorithm and assumptions apply to instrumental variables linear and logistic regression. Moreover, we experimentally validate that our estimator outperforms classical IV regression and two-stage Huber regression on synthetic and semi-synthetic datasets with corruption.
Introduction
The Generalized Method of Moments (GMM) is a fundamental pillar of statistical estimation, particularly within econometrics and causal inference. By identifying parameters through "moment conditions"—equations that should equal zero on average, such as the covariance between an instrument and an error term—GMM provides a flexible framework for a wide array of models.
Despite its versatility, classical GMM is notoriously fragile. In the presence of even a small fraction of adversarial outliers, the resulting estimates can be shifted arbitrarily, leading to incorrect inferences. While robust statistics has addressed similar issues since the 1960s (e.g., via Huber loss), traditional robust GMM methods have historically struggled with three major hurdles: computational intractability in high dimensions, accuracy that degrades as the number of variables increases, and a lack of quantitative recovery guarantees for constant fractions of corruption. This paper introduces the first computationally efficient algorithm that overcomes these limitations.
The ITERATED-GMM-SEVER Algorithm
The authors propose a novel algorithm, ITERATED-GMM-SEVER, which extends the SEVER framework—originally designed for robust stochastic optimization—to the more complex GMM setting. The algorithm relies on two primary mechanisms: spectral filtering and iterative refinement.
Spectral Filtering
Traditional outlier removal often focuses on deleting points with large magnitudes. In contrast, this algorithm employs spectral filtering. It examines the covariance matrix of the moment functions to identify the direction of highest variance (the top eigenvector). By removing samples that contribute most significantly to the variance in this specific direction, the algorithm can prune outliers that are strategically placed to bias the mean without necessarily being "large" in a Euclidean sense.
Two-Stage Filtering
A key technical insight of this work is that in GMM, the moment function $g(X, \theta)$ is not always the gradient of a simple loss function. Consequently, filtering based solely on gradients (as in previous robust optimization methods) is insufficient. The authors implement a two-stage filtering process that considers both the moments themselves and the gradients of those moments. This ensures that the empirical mean of the moments remains close to the true population mean.
Iterative Refinement
To achieve high precision, the algorithm uses an iterative approach. It begins with a coarse estimate and a relatively large search radius. By repeatedly applying the robust sub-routine and subsequently "zooming in"—shrinking the search radius as the estimate improves—the algorithm converges to the true parameter $\theta^*$ with high accuracy.
Theoretical and Empirical Results
Theoretical Guarantees
The paper provides a rigorous $\ell_2$ recovery guarantee of $O(\sqrt{\epsilon})$, where $\epsilon$ is the fraction of corrupted data. Crucially, this bound is dimension-independent, meaning the accuracy does not degrade as the number of features $d$ grows. This makes the method particularly suitable for modern high-dimensional datasets in fields like genomics or social science.
Synthetic IV Regression
In experiments involving 20-dimensional Instrumental Variable (IV) regression with 10,000 samples, the proposed method demonstrated remarkable resilience. Even when 40-50% of the samples were replaced with adversarial noise, the recovery error remained low. In contrast, classical IV regression and "Huberized" variants saw their error rates explode almost immediately upon the introduction of corruption.
Semi-synthetic "Card" Dataset
The algorithm was also tested on a semi-synthetic version of David Card’s 1993 study on the economic returns to schooling. Using "proximity to a 4-year college" as an instrument, the authors showed that an adversary could flip the sign of the estimated return to education (making it appear as though education decreases wages) by corrupting only 15% of the data. The ITERATED-GMM-SEVER algorithm successfully ignored these outliers, recovering the correct positive Average Treatment Effect (ATE).
Limitations and Conclusion
While the algorithm represents a significant leap forward, some limitations remain. The method requires an initial upper bound on the norm of the true parameter (the starting radius), which, while often available, is a formal requirement for the convergence proofs. Additionally, the algorithm can become unstable at very high corruption levels (e.g., >15% in the Card dataset) due to large-norm outliers; the authors suggest a simple norm-thresholding preprocessing step could mitigate this.
By providing the first computationally efficient and provably robust GMM estimator, this work bridges the gap between high-dimensional robust statistics and practical econometrics, offering a powerful tool for reliable causal inference in the presence of data contamination.
Summary updated