Enabling Mixed Effects Neural Networks for Diverse, Clustered Data Using Monte Carlo Methods

Andrej Tschalzev, Paul Nitschke, Lukas Kirchdorfer, Stefan Lüdtke, Christian Bartelt, Heiner Stuckenschmidt

Abstract

Mixed Effects Neural Networks (MENNs) offer a powerful way to model clustered data by separating fixed and random effects. However, existing MENN frameworks struggle with multi-class classification and high-cardinality features due to the lack of closed-form solutions for the likelihood function. We present MC-GMENN, a framework that utilizes Monte Carlo methods—specifically the No-U-Turn Sampler (NUTS)—to decouple random effect sampling from neural network optimization. Our results demonstrate that MC-GMENN achieves state-of-the-art predictive performance, provides unbiased variance estimates, and scales efficiently to large datasets with multiple clustering variables.

Introduction

In many real-world datasets, observations are not independent; they are clustered. For example, medical records are grouped by hospital, and e-commerce transactions are grouped by product brand. Standard Deep Neural Networks (DNNs) usually ignore this structure or attempt to handle it using categorical embeddings. However, embeddings for high-cardinality features (features with thousands of unique categories) often lead to overfitting and lack interpretability.

Mixed Effects Neural Networks (MENNs) were designed to solve this by separating "fixed effects" (patterns that apply globally) from "random effects" (patterns specific to a cluster). The challenge lies in training: for classification, the loss function has no closed-form solution. Previous attempts used biased approximations like Variational Inference or non-scalable methods like Gaussian Quadrature, limiting MENNs to simple regression or binary classification with a single clustering variable.

Methods

MC-GMENN introduces a robust training loop based on the Expectation-Maximization (EM) algorithm. The model architecture combines a standard DNN ($f_\Omega$) for fixed effects with a linear component for random effects ($ZB$).

MC-GMENN Architecture: Decoupling fixed effects (DNN) from random effects (MCMC sampling).

The core innovation lies in the decoupling of the EM steps:

  1. The E-Step (Sampling): Instead of rough approximations, the framework uses the No-U-Turn Sampler (NUTS)—a sophisticated Markov Chain Monte Carlo (MCMC) algorithm—to draw samples of the random effects.
  2. The M-Step (Optimization): Neural network weights are updated using standard Stochastic Gradient Descent (SGD).

Crucially, the sampling process is detached from the mini-batch loss evaluation. The expensive sampling happens once per epoch, while the neural network is trained efficiently using mini-batches. This allows the model to handle multi-class classification and multiple clustering features simultaneously.

Results and Key Insights

The authors evaluated MC-GMENN across simulated binary and multi-class data, as well as 16 real-world benchmarks.

Superior Performance and Efficiency

MC-GMENN consistently outperformed or matched state-of-the-art MENNs (like LMMNN and ARMED) and standard embedding techniques. In multi-class simulations, it achieved a Mean Reciprocal Rank (MRR) of 0.87, compared to 0.44 for embeddings. Despite using MCMC, MC-GMENN was significantly faster than LMMNN—roughly 8x faster in simulations—because it avoids the expensive matrix inversions required by other methods.

Performance comparison showing MC-GMENN's superior accuracy and significantly lower training time compared to baselines.

Unbiased Variance and Interpretability

A major advantage of MC-GMENN is its ability to provide unbiased variance estimates. Previous models using Variational Inference were often biased toward their priors, underestimating the variance between clusters. MC-GMENN provides an accurate quantification of cluster-level influence, which is critical for high-stakes domains like personalized medicine. Furthermore, because the random effects are linear and probabilistic, the model offers "white-box" interpretability, allowing researchers to visualize the distribution of effects for different clusters.

Conclusion

MC-GMENN demonstrates that MCMC is finally viable for large-scale Deep Learning by focusing sampling specifically on random effects. The framework successfully handled datasets with 1 million samples and 100 target classes without a massive explosion in training time. While currently limited to linear random effects, MC-GMENN provides a statistically rigorous and efficient foundation for modeling complex, clustered data structures.

Summary updated

BibTeX
@inproceedings{tschalzev2024enabling,
  author = {Andrej Tschalzev and Paul Nitschke and Lukas Kirchdorfer and Stefan Lüdtke and Christian Bartelt and Heiner Stuckenschmidt},
  title = {{Enabling Mixed Effects Neural Networks for Diverse, Clustered Data Using Monte Carlo Methods}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure