Introduction
Linear Mixed Effects (LME) models are fundamental tools for analyzing longitudinal data, such as tracking patient health or athlete performance over time. However, modern datasets often feature a high number of potential predictors, necessitating variable selection to identify significant factors. Performing variable selection for individual-level random effects is computationally intensive, as traditional Bayesian methods like spike-and-slab priors require exploring a model space that grows exponentially ($2^p$) with the number of predictors. This paper introduces a high-speed Bayesian inference framework designed to make these complex, sparse models practical for large-scale applications.
Methodology
The authors propose an optimization-based approach to replace traditional sampling-based Markov Chain Monte Carlo (MCMC) methods.
Occam’s Window Approximation
The core innovation is the "Occam’s window" approximation. Instead of evaluating all $2^p$ possible models, the algorithm identifies a small subset of high-probability models. By ignoring models significantly less likely than the best-performing one, the computational burden is drastically reduced.
Inference Algorithms
For models with normally distributed errors, an Expectation-Maximization (EM) algorithm is employed, where the E-step uses Occam’s window to approximate the posterior of variable inclusion. For real-world data that often exhibits skewness and heavy tails, the framework is extended to skew-$t$ distributions using a Variational Bayes (VB) scheme. The skew-$t$ distribution is represented as a mixture of normals with latent variables, facilitating the integration of the Occam’s window logic.
Model Search
A greedy search algorithm is used to populate Occam's window. It proposes new models by flipping variable inclusion indicators and retaining those that improve the marginal likelihood.
Experimental Evaluation
The method was evaluated using both simulated and real-world datasets:
- Simulations: 32 combinations of parameters varying in sparsity, skewness, and dimensionality.
- Elite Athlete Performance: Longitudinal data from approximately 1,300 female and 2,800 male sprinters (150,000 observations), and 1,200 female and 1,600 male weightlifters (26,000 observations). The goal was to model career trajectories using linear splines as random effects.
Key Results
The proposed VB algorithm demonstrated a 5 to 6 times speedup over traditional MCMC methods. Despite the approximation, the method maintained high accuracy. In simulations, the True Positive Rate (TPR) remained above 0.92 across nearly all settings, indicating effective identification of underlying predictors.
In real-world applications, the VB estimates for skewness and degrees of freedom closely matched those from MCMC, validating the approximation's reliability. Furthermore, the algorithm successfully scaled to datasets with over 95,000 observations.
Conclusion and Future Work
This framework enables individualized sparsity, allowing different variables to be selected for different subjects—a critical feature in fields like sports science. While the greedy search is efficient, future work may address its potential to miss the global optimum in extremely high-dimensional spaces. Additionally, extending this framework to other non-normal distributions, such as the skew-normal, could broaden its utility in insurance and medicine.
Summary updated