SLIM: Stochastic Learning and Inference in Overidentified Models

Xiaohong Chen, Min Seong Kim, Sokbae Lee, Myung Hwan Seo, Myung-Hyun SongarXiv 2025 · DOI · arXiv

Abstract

We propose SLIM (Stochastic Learning and Inference in overidentified Models), a scalable stochastic approximation framework for nonlinear GMM. SLIM forms iterative updates from independent mini-batches of moments and their derivatives, producing unbiased directions that ensure almost-sure convergence. It requires neither a consistent initial estimator nor global convexity and accommodates both fixed-sample and random-sampling asymptotics. We further develop an optional second-order refinement achieving full-sample GMM efficiency and inference procedures based on random scaling and plug-in methods, including plug-in, debiased plug-in, and online versions of the Sargan--Hansen $J$-test tailored to stochastic learning. In Monte Carlo experiments based on a nonlinear demand system with 576 moment conditions, 380 parameters, and $n = 10^5$, SLIM solves the model in under 1.4 hours, whereas full-sample GMM in Stata on a powerful laptop converges only after 18 hours. The debiased plug-in $J$-test delivers satisfactory finite-sample inference, and SLIM scales smoothly to $n = 10^6$.

Introduction

In modern econometrics, the Generalized Method of Moments (GMM) is a cornerstone for estimating models defined by moment conditions. However, as datasets scale into the millions of observations, traditional GMM becomes computationally prohibitive. Standard optimization requires evaluating the entire dataset at every iteration, creating a "memory wall" for researchers. While machine learning has addressed similar scaling issues through Stochastic Gradient Descent (SGD), GMM presents a unique challenge: the gradient of its objective function involves a product of sample averages. Using a single mini-batch to estimate this gradient introduces a fundamental bias that prevents standard SGD from converging to true parameter values.

The SLIM Framework

SLIM (Stochastic Learning and Inference in overidentified Models) overcomes this bias through a novel U-statistic approach. By utilizing independent mini-batches to create unbiased update directions, SLIM enables researchers to perform estimation and valid statistical inference on massive datasets that would otherwise be infeasible.

The SLIM workflow: from initial learning to statistical inference.

The core innovation lies in the independence of data streams. At each iteration, SLIM draws two independent mini-batches—one to estimate the moment conditions and another to estimate their Jacobian derivatives. This independence ensures the update direction is unbiased. To stabilize the estimation path and achieve optimal convergence rates, the algorithm employs Polyak-Ruppert averaging, maintaining a running average of all previous estimates.

Modular Architecture and Inference

SLIM is designed as a modular two-step process:

  1. Step 1 (Required): A first-order approach that is robust to poor initialization. Unlike prior "online GMM" methods, SLIM does not require a "warm start" or a consistent initial estimator from a large data chunk; it can converge almost surely starting from zero.
  2. Step 2 (Optional): A second-order refinement that incorporates an optimal weighting matrix. This step ensures the estimator achieves the same statistical efficiency as full-sample GMM.

For statistical inference, SLIM provides two primary paths. Random Scaling is an online method that updates uncertainty estimates dynamically, avoiding the need to store or invert massive covariance matrices. Alternatively, a Plug-in approach can be used when the full dataset is eventually accessible. The framework also introduces adapted versions of the Sargan–Hansen $J$-test, including "online" and "debiased" variants, to verify model specification in a stochastic setting.

Empirical Performance

The researchers evaluated SLIM using Monte Carlo experiments based on the Exact Affine Stone Index (EASI) demand model, featuring 576 moment conditions and 380 parameters.

In tests with $n = 100,000$ observations, SLIM solved the model in under 1.4 hours. In contrast, standard GMM implemented in Stata on a high-performance laptop required 18 hours to reach the same result. Furthermore, SLIM scaled smoothly to $n = 1,000,000$, a regime where traditional full-sample GMM is practically infeasible due to memory and time constraints. The debiased plug-in $J$-test and random scaling confidence intervals demonstrated excellent performance, maintaining correct size while remaining powerful enough to detect model misspecification.

Conclusion and Future Work

SLIM brings the scalability of stochastic optimization to the rigorous world of nonlinear GMM. While currently requiring smooth, twice-differentiable moment functions, future extensions may adapt the framework for non-smooth models like quantile regressions or dependent data structures such as time-series. By bypassing the computational bottlenecks of traditional econometrics, SLIM opens the door for complex structural modeling on truly massive datasets.

Summary updated

Abstract

We propose SLIM (Stochastic Learning and Inference in overidentified Models), a scalable stochastic approximation framework for nonlinear GMM. SLIM forms iterative updates from independent mini-batches of moments and their derivatives, producing unbiased directions that ensure almost-sure convergence. It requires neither a consistent initial estimator nor global convexity and accommodates both fixed-sample and random-sampling asymptotics. We further develop an optional second-order refinement achieving full-sample GMM efficiency and inference procedures based on random scaling and plug-in methods, including plug-in, debiased plug-in, and online versions of the Sargan--Hansen $J$-test tailored to stochastic learning. In Monte Carlo experiments based on a nonlinear demand system with 576 moment conditions, 380 parameters, and $n = 10^5$, SLIM solves the model in under 1.4 hours, whereas full-sample GMM in Stata on a powerful laptop converges only after 18 hours. The debiased plug-in $J$-test delivers satisfactory finite-sample inference, and SLIM scales smoothly to $n = 10^6$.
BibTeX
@misc{chen2025slim,
  author = {Xiaohong Chen and Min Seong Kim and Sokbae Lee and Myung Hwan Seo and Myung-Hyun Song},
  title = {{SLIM: Stochastic Learning and Inference in Overidentified Models}},
  year = {2025},
  eprint = {2510.20996},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.20996},
  howpublished = {arXiv},
}
Figure