Estimating the Random Effect in Big Data Mixed Models

Michael Law, Ya’acov RitovarXiv 2019 · DOI · arXiv

Abstract

In high-dimensional Gaussian linear mixed models (LMMs), performing reliable statistical inference remains a significant challenge when the number of fixed-effects covariates exceeds the sample size. This paper addresses three critical problems in this domain: constructing an asymptotic $F$-statistic for testing the presence of random effects, deriving $\sqrt{n}$-rate asymptotic confidence intervals for individual random effects, and proposing an empirical Bayes estimator for ANOVA-type models. By "retrofitting" classical low-dimensional techniques with high-dimensional tools such as exponential weighting, the proposed methods achieve performance comparable to oracle estimators. The validity of these approaches is demonstrated through numerical simulations and an application to the Trends in International Mathematics and Sciences Study (TIMSS) dataset.

Introduction

Linear Mixed Models (LMMs) are a cornerstone of statistical analysis, allowing researchers to account for both systematic relationships (fixed effects) and random variations (random effects) within structured data. While high-dimensional linear models—where the number of features $p$ is much larger than the number of observations $n$—have seen extensive development through methods like the Lasso, their extension to mixed models has been less explored.

In "Big Data" contexts, traditional LMM inference often breaks down. Standard methods for testing the existence of random effects or estimating variance components typically require restrictive assumptions about the design matrix or the relationship between covariates. This paper introduces a robust framework for performing inference in high-dimensional LMMs without demanding a "perfect" design for the fixed effects.

Methodology: The Retrofitting Approach

The authors propose a "retrofitting" strategy that adapts well-understood low-dimensional statistical procedures for high-dimensional settings. This is achieved through two primary mechanisms:

1. Exponential Weighting (EW)

Rather than selecting a single "best" sparse model (as in Lasso regression), the authors utilize exponential weighting to aggregate results across a collection of potential models. This Bayesian-inspired approach provides a more stable prediction of the mean vector. Crucially, it remains robust even when the fixed-effects design matrix fails to satisfy strict conditions like the "compatibility condition," which is often required for the consistency of sparse estimators.

2. Projection and Whitening

To isolate the random effects from the high-dimensional fixed-effects "nuisance" parameters, the data is projected into specific subspaces. The authors then employ spectral decomposition to "whiten" the noise. This transformation simplifies the data structure, making it possible to construct valid test statistics and confidence intervals that behave predictably as the sample size grows.

Statistical Inference and Estimation

The framework yields three primary tools for the practitioner:

  • Asymptotic F-test: A robust test to determine if a collection of random effects is significantly different from zero. This is vital for model selection and for determining if group-level variations (e.g., school-level or country-level differences) are statistically meaningful.
  • Parametric Rate Confidence Intervals: The authors derive confidence intervals for the variance of random effects ($\sigma^2_\nu$) that converge at the ideal $\sqrt{n}$ rate. This demonstrates that high dimensionality does not inherently prevent the achievement of classical parametric convergence speeds.
  • Empirical Bayes Estimator: For ANOVA-type models, the authors propose an estimator for group-level means. They prove that this estimator performs asymptotically as well as an "oracle" Bayes estimator—one that assumes the true variance parameters are known a priori.

Experimental Validation

Synthetic Simulations

The methods were tested using simulated datasets where $n=200$ and $p=500$. The results confirmed that the proposed $F_{EW}$ test maintains the intended 5% Type I error rate (correctly identifying when random effects are zero) and exhibits high power in detecting small random effects as $n$ increases. Furthermore, the joint estimation of the mean vector across different data projections was shown to outperform separate treatments by effectively pooling information.

TIMSS 2015 Application

The procedures were applied to the Trends in International Mathematics and Sciences Study (TIMSS) data, analyzing math scores across 6,808 schools in 43 countries. Using 106 background covariates as fixed effects, the model successfully identified significant country-level impacts on math ability. The resulting rankings of the top-performing countries—South Korea, Hong Kong, Singapore, Chinese Taipei, and Japan—aligned closely with official international reports, validating the model's real-world utility.

Discussion and Limitations

While this framework provides a powerful new toolkit for high-dimensional LMMs, several avenues for future research remain. The current theory relies on the assumption of Gaussian (normal) distributions for both errors and random effects. While standard in the field, extending these results to non-Gaussian or "heavy-tailed" data would increase the model's robustness.

Additionally, the computational cost of exponential weighting, which often requires Metropolis-Hastings sampling, can be significant for extremely large datasets. Finally, while the fixed effects are high-dimensional, the current framework assumes the random effects themselves remain relatively low-dimensional ($v + r < n$). Expanding this methodology to accommodate high-dimensional random effects is a promising direction for future work.

Summary updated

Abstract

We consider three problems in high-dimensional Gaussian linear mixed models. Without any assumptions on the design for the fixed effects, we construct an asymptotic $F$-statistic for testing whether a collection of random effects is zero, derive an asymptotic confidence interval for a single random effect at the parametric rate $\sqrt{n}$, and propose an empirical Bayes estimator for a part of the mean vector in ANOVA type models that performs asymptotically as well as the oracle Bayes estimator. We support our results with numerical simulations and provide comparisons with oracle estimators. The procedures developed are applied to the Trends in International Mathematics and Sciences Study (TIMSS) data.
BibTeX
@misc{law2019estimating,
  author = {Michael Law and Ya’acov Ritov},
  title = {{Estimating the Random Effect in Big Data Mixed Models}},
  year = {2019},
  eprint = {1907.11958},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.11958},
}
Figure