Likelihood ratio test for variance components in nonlinear mixed effects models

Charlotte Baey, Paul-Henry Cournède, Estelle KuhnarXiv 2017 · DOI · arXiv

Abstract

Mixed effects models are widely used to describe heterogeneity in a population by incorporating both fixed and random effects. A crucial challenge in refining these models is identifying which parameters should be treated as random—specifically, testing whether the variances of a given subset of random effects are zero. While the Likelihood Ratio Test (LRT) is a standard tool for model comparison, its application to variance components is complicated by the fact that the null hypothesis lies on the boundary of the parameter space. This paper establishes the asymptotic properties of the LRT for testing zero-variance components in both linear and nonlinear mixed effects models (NLMEMs). We prove that the limiting distribution is a chi-bar-square ($\bar{\chi}^2$) distribution—a mixture of chi-square distributions—and identify the corresponding weights. Notably, we demonstrate that this distribution depends on the correlation structure between random effects but is independent of the model's linear or nonlinear structure. The procedure is validated through simulation studies and applied to datasets involving dental growth and coucal growth rates.

Introduction

In biological and physical modeling, mixed effects models allow researchers to account for population-wide trends while acknowledging individual-level variations. However, determining whether a specific parameter truly varies across a population or remains constant is a non-trivial statistical task. Formally, this involves testing if the variance of a random effect is zero. If the variance is zero, the parameter is effectively a fixed effect.

In standard statistical contexts, the Likelihood Ratio Test (LRT) statistic typically follows a chi-square ($\chi^2$) distribution. However, this asymptotic behavior relies on the assumption that the parameter under the null hypothesis lies in the interior of the parameter space. Because variances cannot be negative, testing for a zero variance places the parameter on the boundary, violating standard regularity conditions. While this "boundary problem" has been explored in linear models, this research provides a rigorous extension to the nonlinear mixed effects models (NLMEMs) frequently used in pharmacology and ecology.

Theoretical Framework and Methodology

To address the boundary problem in NLMEMs, the authors utilized tools from convex analysis, specifically approximating the parameter space near the boundary using tangent cones.

The Asymptotic Distribution

The study proves that under specific regularity conditions, the LRT statistic for NLMEMs converges to a chi-bar-square ($\bar{\chi}^2$) distribution. This distribution is defined as a weighted mixture of chi-square distributions with varying degrees of freedom:

$$P(\bar{\chi}^2(C, V) \leq x) = \sum_{i=0}^q w_i(C, V) P(\chi^2_i \leq x)$$

where $w_i$ represents the weights, $C$ is the cone approximating the parameter space, and $V$ is the asymptotic covariance matrix of the maximum likelihood estimator.

Identifying Weights

A significant contribution of this work is the identification of the weights for the $\bar{\chi}^2$ mixture. These weights are determined by the Fisher Information Matrix and the correlation structure of the random effects. The authors highlight that if random effects are correlated, the weights shift significantly, requiring a more complex calculation than the simple 50/50 mixtures often assumed in simpler cases.

Numerical Implementation

Because NLMEMs often lack closed-form likelihoods, the authors proposed a computational framework using the Stochastic Approximation EM (SAEM) algorithm combined with Monte Carlo integration. This allows for the estimation of test statistics and p-values even in complex nonlinear settings where direct integration is impossible.

Key Insights

  1. Invariance to Model Structure: One of the most striking findings is that the asymptotic distribution of the LRT is independent of whether the model is linear or nonlinear. The "nonlinear-ness" does not alter the limiting distribution, provided the regularity conditions are met.
  2. The Role of Correlations: While the model structure may not matter, the correlation between random effects is critical. Ignoring correlations when they exist leads to incorrect weight calculations and, consequently, erroneous p-values.
  3. Boundary Geometry: The use of tangent cones provides a robust mathematical way to handle various constraints on the parameter space, offering a generalized approach to variance component testing.

Empirical Validation and Applications

Simulation Studies

The authors conducted simulations using linear and logistic growth models. The results confirmed that as the sample size ($N$) increases, the empirical level of the test (the false positive rate) aligns with the theoretical nominal level predicted by the $\bar{\chi}^2$ distribution.

Dental Growth Data

Using the classic Potthoff and Roy (1964) dataset, which tracks dental measurements in children, the authors tested whether the "slope" of growth should be treated as a random effect. Interestingly, the conclusion depended on the correlation assumption: assuming independence between the intercept and slope led to a rejection of the null hypothesis ($p=0.028$), whereas accounting for correlation resulted in a higher p-value ($p=0.082$), suggesting the variance might not be significantly different from zero at the 5% level.

Coucal Growth Rates

The test was also applied to a nonlinear model of body mass growth in white-browed coucal nestlings. The LRT correctly identified that certain growth parameters did not require random effects, allowing for a more parsimonious model without sacrificing explanatory power.

Conclusion

This research provides a definitive mathematical foundation for testing variance components in nonlinear mixed effects models. By proving the $\bar{\chi}^2$ asymptotic distribution and emphasizing the importance of correlation structures, the authors offer a reliable roadmap for practitioners. While the computational cost of Monte Carlo simulations remains a factor for high-dimensional models, this work ensures that researchers can perform these critical tests with statistical confidence, avoiding the pitfalls of the "boundary problem."

Summary updated

Abstract

Mixed effects models are widely used to describe heterogeneity in a population. A crucial issue when adjusting such a model to data consists in identifying fixed and random effects. From a statistical point of view, it remains to test the nullity of the variances of a given subset of random effects. Some authors have proposed to use the likelihood ratio test and have established its asymptotic distribution in some particular cases. Nevertheless, to the best of our knowledge, no general variance components testing procedure has been fully investigated yet. In this paper, we study the likelihood ratio test properties to test that the variances of a general subset of the random effects are equal to zero in both linear and nonlinear mixed effects model, extending the existing results. We prove that the asymptotic distribution of the test is a chi-bar-square distribution, that is to say a mixture of chi-square distributions, and we identify the corresponding weights. We highlight in particular that the limiting distribution depends on the presence of correlations between the random effects but not on the linear or nonlinear structure of the mixed effects model. We illustrate the finite sample size properties of the test procedure through simulation studies and apply the test procedure to two real datasets of dental growth and of coucal growth.
BibTeX
@misc{baey2017likelihood,
  author = {Charlotte Baey and Paul-Henry Cournède and Estelle Kuhn},
  title = {{Likelihood ratio test for variance components in nonlinear mixed effects models}},
  year = {2017},
  eprint = {1712.08567},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1712.08567},
}
Figure