SGMM: Stochastic Approximation to Generalized Method of Moments

Xiaohong Chen, Sokbae Lee, Yuan Liao, Myung Hwan Seo, Youngki Shin, Myunghyun Song

TL;DR

This paper introduces SGMM, a new class of online algorithms for estimating and testing models with moment restrictions, a common task in econometrics. SGMM offers a computationally efficient and scalable alternative to traditional offline methods, making it suitable for large and streaming datasets.

What Problem Does This Paper Address?

Econometric models often rely on "moment restrictions," which are conditions that specify expected values of certain functions of the data and model parameters. The Generalized Method of Moments (GMM) is a popular framework for estimating parameters under these restrictions, especially when there are more restrictions than parameters (overidentification). However, traditional GMM methods are "offline," meaning they require the entire dataset to be available before estimation. This is a significant limitation when dealing with massive datasets or real-time data streams, where processing the entire dataset is computationally infeasible or impossible. The paper addresses this gap by developing an online, stochastic approximation-based approach to GMM.

Methods

The core contribution is the development of Stochastic Generalized Method of Moments (SGMM). This is a novel stochastic approximation (SA) algorithm, building on the principles of stochastic gradient descent (SGD).

Key methodological aspects include:

  • Stochastic Approximation for GMM: Unlike standard SGD which is typically used for minimizing loss functions (M-estimation), SGMM is designed for Z-estimation problems, where parameters are identified by moment conditions.
  • Online 2SLS and Efficient SGMM: The paper first proposes an online version of Two-Stage Least Squares (2SLS) and then an efficient SGMM estimator. These algorithms update parameter estimates sequentially, one observation at a time.
  • Optimal Weighting Matrix Update: A crucial aspect of efficient GMM is the use of an optimally weighted objective function. SGMM incorporates an online update for this weighting matrix, leveraging the Sherman-Morrison-Woodbury (SMW) formula for computational efficiency.
  • Online Inference and Tests: SGMM is extended to provide online versions of statistical inference tools, including confidence intervals and hypothesis tests like the Durbin-Wu-Hausman (DWH) and Sargan-Hansen tests.
  • Theoretical Guarantees: The authors establish theoretical properties for their algorithms, including almost sure convergence and functional central limit theorems (FCLTs) for both the inefficient 2SLS and the efficient SGMM estimators.

Datasets

The paper uses two well-known empirical examples with large sample sizes for demonstration:

  • Angrist and Krueger (1991): This study examines the return to education.
  • Angrist and Evans (1998): This study investigates the effect of childbearing on female labor supply.

The paper does not introduce new benchmark datasets.

Key Insights and Ideas

  • Bridging the Gap between ML and Econometrics: SGMM effectively translates the computational advantages of online machine learning techniques (like SGD) into the econometric domain, addressing a long-standing challenge for large-scale moment restriction models.
  • Scalability for Big Data: The online nature of SGMM is its primary advantage, enabling real-time estimation and inference on datasets that would overwhelm traditional offline GMM.
  • Efficiency Maintained Online: A significant insight is that SGMM not only achieves computational efficiency but also matches the estimation accuracy of offline GMM, particularly as sample sizes grow.
  • Online Inference is Feasible: The paper demonstrates that complex econometric inference (confidence intervals, hypothesis tests) can be performed online, which was previously a major hurdle.
  • Robustness to Overidentification: SGMM is designed to handle overidentified models, a common scenario in econometrics, and integrates online versions of tests for overidentifying restrictions.

Key Results

  • Asymptotic Equivalence: The proposed SGMM estimator is shown to be first-order asymptotically equivalent to the standard offline efficient GMM estimator.
  • Computational Speedup: Monte Carlo simulations show substantial gains in computation time for SGMM compared to offline methods (2SLS, GMM), especially for large sample sizes. For instance, GMM was up to 7.6 times slower than SGMM when n > 10⁶ in one experiment.
  • Accuracy: SGMM achieves comparable estimation accuracy (RMSE, Bias, SD) to offline GMM, particularly for large sample sizes (n ≥ 10⁶).
  • Coverage Probabilities: SGMM and its variants demonstrate good coverage probabilities for confidence intervals across various simulation designs.
  • Empirical Performance: In empirical applications, SGMM estimators were significantly faster than offline GMM, with comparable or improved confidence interval lengths.

Limitations and Future Work

  • Nonlinear SGMM: The paper mentions an extension to nonlinear SGMM as a topic for future research.
  • Weak Instruments: The current framework assumes valid and strong instruments. Extending SGMM to handle many weak instruments in nonlinear GMM models is an important open question.
  • Multi-Epoch Algorithms: While the paper briefly explores multi-epoch approaches for empirical examples, a formal theoretical extension to this setting and developing early stopping rules are suggested for future work.
  • Computational Demands for Inference: While SGMM is computationally efficient for estimation, some online inference procedures (like exact forms of DWH/Sargan-Hansen tests) can still be computationally demanding, leading the authors to adopt robust approximations.

Summary updated

Abstract

We introduce a new class of algorithms, Stochastic Generalized Method of Moments (SGMM), for estimation and inference on (overidentified) moment restriction models. Our SGMM is a novel stochastic approximation alternative to the popular Hansen (1982) (offline) GMM, and offers fast and scalable implementation with the ability to handle streaming datasets in real time. We establish the almost sure convergence, and the (functional) central limit theorem for the inefficient online 2SLS and the efficient SGMM. Moreover, we propose online versions of the Durbin-Wu-Hausman and Sargan-Hansen tests that can be seamlessly integrated within the SGMM framework. Extensive Monte Carlo simulations show that as the sample size increases, the SGMM matches the standard (offline) GMM in terms of estimation accuracy and gains over computational efficiency, indicating its practical value for both large-scale and online datasets. We demonstrate the efficacy of our approach by a proof of concept using two well known empirical examples with large sample sizes.
BibTeX
@misc{chen2023sgmm,
  author = {Xiaohong Chen and Sokbae Lee and Yuan Liao and Myung Hwan Seo and Youngki Shin and Myunghyun Song},
  title = {{SGMM: Stochastic Approximation to Generalized Method of Moments}},
  year = {2023},
  eprint = {2308.13564},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.13564},
}
Figure