Method-of-Moments Inference for GLMs and Doubly Robust Functionals under Proportional Asymptotics

Xingyu Chen, Lin Liu, Rajarshi MukherjeearXiv 2024 · DOI · arXiv

Abstract

In this paper, we consider the estimation of regression coefficients and signal-to-noise (SNR) ratio in high-dimensional Generalized Linear Models (GLMs), and explore their implications in inferring popular estimands such as average treatment effects in high-dimensional observational studies. Under the "proportional asymptotic" regime and Gaussian covariates with known (population) covariance $\Sigma$, we derive Consistent and Asymptotically Normal (CAN) estimators of our targets of inference through a Method-of-Moments type of estimators that bypasses estimation of high dimensional nuisance functions and hyperparameter tuning altogether. Additionally, under non-Gaussian covariates, we demonstrate universality of our results under certain additional assumptions on the regression coefficients and $\Sigma$. We also demonstrate that knowing $\Sigma$ is not essential to our proposed methodology when the sample covariance matrix estimator is invertible. Finally, we complement our theoretical results with numerical experiments and comparisons with existing literature.

Introduction

Classical statistical inference typically assumes a regime where the sample size $n$ is significantly larger than the number of features $p$. However, modern datasets frequently operate in a high-dimensional "proportional asymptotic" regime where $p \approx n$. In such settings, standard Maximum Likelihood Estimation (MLE) often yields biased or inconsistent results.

Existing solutions generally rely on "debiasing" techniques or strict sparsity assumptions (assuming most features have zero effect). However, these methods often require complex estimation of nuisance functions (like propensity scores) and sensitive hyperparameter tuning (e.g., selecting $\lambda$ in Lasso). This paper proposes a simpler, more robust alternative: a Method-of-Moments (MoM) framework that provides $\sqrt{n}$-consistent inference for low-dimensional summaries—such as treatment effects or signal strength—without requiring sparsity or tuning.

Methods and Theoretical Framework

The core innovation is the identification of a set of data moments that can be mapped directly to the parameters of interest. The methodology rests on several key pillars:

  1. Diffeomorphism Mapping: The authors establish a smooth, one-to-one mapping between observable data moments (such as average outcomes and specific correlations) and target parameters like the quadratic form $\|\beta\|_\Sigma^2$.
  2. U-Statistics: Second-order and higher-order U-statistics are employed to create unbiased estimators of these moments, ensuring the final estimates are robust.
  3. Universality: While initially derived for Gaussian designs, the framework is extended to non-Gaussian distributions (e.g., Rademacher) through "universality" principles, provided the signal is delocalized across features.
  4. Handling Unknown Covariance: The paper addresses scenarios where the population covariance $\Sigma$ is unknown, utilizing sample splitting for $p < n$ and Chebyshev polynomial approximations for the $p \geq n$ case.

Assumptions

To establish these theoretical guarantees, the paper defines a set of assumptions ranging from global constraints on the design to specific conditions for universality.

Summary of assumptions used throughout the paper and their respective scopes.

Key Insights

  • Bypassing Nuisance Functions: In observational studies, the MoM approach allows researchers to skip explicit modeling of propensity scores while still achieving consistent estimates of treatment effects.
  • Plug-and-Play Inference: Unlike Ridge or Lasso regression, the MoM estimator does not require cross-validation. It is a direct calculation followed by an inversion of the mapping.
  • Inconsistency as a Tool: The paper demonstrates that even when the full high-dimensional vector $\beta$ cannot be consistently estimated, valid and powerful inference is still possible for its low-dimensional functionals.

Results and Numerical Validation

The authors validated their theory through extensive numerical simulations using both synthetic Gaussian and Rademacher covariates. They tested both dense (non-sparse) and sparse configurations, comparing the MoM estimator against state-of-the-art debiasing methods.

Theoretical Guarantees: The estimators were proven to be Consistent and Asymptotically Normal (CAN). This allows for the construction of valid p-values and confidence intervals in high-dimensional settings.

Performance: Simulations indicated that the MoM estimator often exhibits lower bias and Mean Squared Error (MSE) than competing Lasso or Ridge-based methods. This advantage is largely attributed to the absence of "tuning error," as the MoM approach does not rely on hyperparameter selection.

Limitations and Future Directions

While robust, the method faces challenges when signal strength (|\beta|) is extremely weak, which can lead to U-statistics yielding non-physical (negative) values for variance parameters in small samples. Additionally, the current framework assumes a known link function. Future research aims to extend this to Single-Index Models (SIM) where the link function is unknown. Finally, the computational complexity of higher-order U-statistics in the $p > n$ regime with unknown covariance remains an area for optimization.

Summary updated

Abstract

In this paper, we consider the estimation of regression coefficients and signal-to-noise (SNR) ratio in high-dimensional Generalized Linear Models (GLMs), and explore their implications in inferring popular estimands such as average treatment effects in high-dimensional observational studies. Under the ``proportional asymptotic'' regime and Gaussian covariates with known (population) covariance $Σ$, we derive Consistent and Asymptotically Normal (CAN) estimators of our targets of inference through a Method-of-Moments type of estimators that bypasses estimation of high dimensional nuisance functions and hyperparameter tuning altogether. Additionally, under non-Gaussian covariates, we demonstrate universality of our results under certain additional assumptions on the regression coefficients and $Σ$. We also demonstrate that knowing $Σ$ is not essential to our proposed methodology when the sample covariance matrix estimator is invertible. Finally, we complement our theoretical results with numerical experiments and comparisons with existing literature.
BibTeX
@misc{chen2024method,
  author = {Xingyu Chen and Lin Liu and Rajarshi Mukherjee},
  title = {{Method-of-Moments Inference for GLMs and Doubly Robust Functionals under Proportional Asymptotics}},
  year = {2024},
  eprint = {2408.06103},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.06103},
}

Loading related content...

Figure