Fast Bayesian inference in a class of sparse linear mixed effects models

Maria-Zafeiria Spyropoulou, James G. Hopker, Jim E. Griffin

Introduction

Linear Mixed Effects (LME) models are fundamental tools for analyzing longitudinal data, such as tracking patient health or athlete performance over time. However, modern datasets often feature a high number of potential predictors, necessitating variable selection to identify significant factors. Performing variable selection for individual-level random effects is computationally intensive, as traditional Bayesian methods like spike-and-slab priors require exploring a model space that grows exponentially ($2^p$) with the number of predictors. This paper introduces a high-speed Bayesian inference framework designed to make these complex, sparse models practical for large-scale applications.

Methodology

The authors propose an optimization-based approach to replace traditional sampling-based Markov Chain Monte Carlo (MCMC) methods.

Occam’s Window Approximation

The core innovation is the "Occam’s window" approximation. Instead of evaluating all $2^p$ possible models, the algorithm identifies a small subset of high-probability models. By ignoring models significantly less likely than the best-performing one, the computational burden is drastically reduced.

Inference Algorithms

For models with normally distributed errors, an Expectation-Maximization (EM) algorithm is employed, where the E-step uses Occam’s window to approximate the posterior of variable inclusion. For real-world data that often exhibits skewness and heavy tails, the framework is extended to skew-$t$ distributions using a Variational Bayes (VB) scheme. The skew-$t$ distribution is represented as a mixture of normals with latent variables, facilitating the integration of the Occam’s window logic.

A greedy search algorithm is used to populate Occam's window. It proposes new models by flipping variable inclusion indicators and retaining those that improve the marginal likelihood.

Experimental Evaluation

The method was evaluated using both simulated and real-world datasets:

  • Simulations: 32 combinations of parameters varying in sparsity, skewness, and dimensionality.
  • Elite Athlete Performance: Longitudinal data from approximately 1,300 female and 2,800 male sprinters (150,000 observations), and 1,200 female and 1,600 male weightlifters (26,000 observations). The goal was to model career trajectories using linear splines as random effects.

Key Results

The proposed VB algorithm demonstrated a 5 to 6 times speedup over traditional MCMC methods. Despite the approximation, the method maintained high accuracy. In simulations, the True Positive Rate (TPR) remained above 0.92 across nearly all settings, indicating effective identification of underlying predictors.

True Positive Rate (TPR) performance across various skew-t distribution parameters (c and f) and model dimensions (p and h).

In real-world applications, the VB estimates for skewness and degrees of freedom closely matched those from MCMC, validating the approximation's reliability. Furthermore, the algorithm successfully scaled to datasets with over 95,000 observations.

Conclusion and Future Work

This framework enables individualized sparsity, allowing different variables to be selected for different subjects—a critical feature in fields like sports science. While the greedy search is efficient, future work may address its potential to miss the global optimum in extremely high-dimensional spaces. Additionally, extending this framework to other non-normal distributions, such as the skew-normal, could broaden its utility in insurance and medicine.

Summary updated

Abstract

Linear mixed effects models are widely used in statistical modelling. We consider a mixed effects model with Bayesian variable selection in the random effects using spike-and-slab priors and develop a optimisation-based inference schemes that can be applied to large data sets. An EM algorithm is proposed for the model with normal errors where the posterior distribution of the variable inclusion parameters is approximated using an Occam’s window approach. Placing this approach within a variational Bayes scheme allows the algorithm to be extended to the model with skew-t errors. The performance of the algorithm is evaluated in a simulation study and applied to a longitudinal model for elite athlete performance in 100 metres track sprinting and weightlifting.
BibTeX
@article{spyropoulou2025fast,
  author = {Maria-Zafeiria Spyropoulou and James G. Hopker and Jim E. Griffin},
  title = {{Fast Bayesian inference in a class of sparse linear mixed effects models}},
  journal = {Stat. Comput.},
  year = {2025},
  doi = {10.1007/s11222-025-10628-4},
}
Figure