Abstract
We study maximum likelihood estimators (MLEs) for the residual variance, the signal-to-noise ratio, and other variance parameters in high-dimensional linear models. These parameters are essential in many statistical applications involving regression diagnostics, inference, tuning parameter selection for high-dimensional regression, and other applications, including genetics. The estimators that we study are not new, and have been widely used for variance component estimation in linear random-effects models. However, our analysis is new and it implies that the MLEs, which were devised for random-effects models, may also perform very well in high-dimensional linear models with fixed-effects, which are more commonly studied in some areas of high-dimensional statistics. The MLEs are shown to be consistent and asymptotically normal in fixed-effects models with random design, in asymptotic settings where the number of predictors ($p$) is proportional to the number of observations ($n$). Moreover, the estimators’ asymptotic variance can be given explicitly in terms moments of the Marcenko-Pastur distribution. A variety of analytical and empirical results show that the MLEs outperform other, previously proposed estimators for variance parameters in high-dimensional linear models with fixed-effects. More broadly, the results in this paper illustrate a strategy for drawing connections between fixed- and random-effects models in high dimensions, which may be useful in other applications.
Introduction
In modern statistical analysis, high-dimensional regimes where the number of predictors ($p$) is large relative to the number of observations ($n$) are increasingly common. Estimating variance parameters—specifically the residual noise ($\sigma^2$) and the signal-to-noise ratio (SNR)—is a fundamental challenge in these settings. Accurate estimation is vital for hypothesis testing, model selection, and applications such as calculating heritability in genetics.
Existing methodologies typically fall into two categories: those assuming signal sparsity (e.g., Scaled-Lasso) and those assuming specific structures for the design matrix $X$. While Maximum Likelihood Estimators (MLEs) are standard for random-effects models, their performance in high-dimensional fixed-effects models has lacked rigorous theoretical grounding. This paper bridges this gap, demonstrating that "old" MLE tools can outperform modern high-dimensional estimators.
Methods
The research focuses on a bivariate MLE for the residual variance ($\sigma_0^2$) and the signal-to-noise ratio ($\eta_0^2$). The authors introduce a novel coupling argument, proving that the distribution of the MLE in a fixed-effects model (constant $\beta$) is identical to its distribution in a random-effects model (random $\beta$) under Gaussian designs. This equivalence allows for the application of Random Matrix Theory to derive the asymptotic properties of the estimators. Specifically, the asymptotic variance is expressed through the moments of the Marčenko-Pastur distribution, which characterizes the eigenvalue behavior of large random matrices.
Key Results and Discussion
The theoretical findings are validated through extensive simulations and real-world data analysis.
Robustness to Sparsity
A significant advantage of the MLE is its independence from signal sparsity. While methods like the Scaled-Lasso and RCV-Lasso exhibit significant bias when the signal is not extremely sparse, the MLE remains unbiased and efficient across a wide range of sparsity levels.
Efficiency Gains
Theoretical analysis reveals that the MLE is more efficient than the Method-of-Moments (MM) and EigenPrism, particularly as the signal-to-noise ratio increases. The asymptotic variance of the MLE is consistently lower than that of the MM estimator across various ratios of $p/n$.
Empirical Performance
In simulations with $n=500$ and $p=1000$, the MLE consistently outperformed MM and EigenPrism in terms of variability. Furthermore, application to genetics data from the International HapMap Project showed that the MLE produced shorter confidence intervals for heritability estimates compared to other methods, indicating higher precision in real-world inference.
Conclusion
This study demonstrates that classical MLEs, originally intended for random-effects models, are superior for variance estimation in high-dimensional fixed-effects models. Their robustness to sparsity and high efficiency make them a powerful tool for modern statistical applications. Future work will aim to relax the Gaussian assumption for the design matrix and address settings with highly correlated predictors.
Summary updated