EMJMCMC: Evolutionary Mode Jumping Markov Chain Monte Carlo Expert Toolbox

Aliaksandr Hubin, Waldir LeoncioOpenalex Unknown 2024 · DOI · OpenAlex

Abstract

05316>, which allow for estimating posterior model probabilities and Bayesian model averaging across a wide set of Bayesian models including linear, generalized linear, generalized linear mixed, generalized nonlinear, generalized nonlinear mixed, and logic regression models.

Introduction

In Bayesian statistics, Bayesian Model Averaging (BMA) is a critical technique for accounting for model uncertainty by averaging predictions across an ensemble of potential models. However, as the number of covariates increases, the model space expands exponentially, making exhaustive search impossible. Traditional Markov Chain Monte Carlo (MCMC) methods often struggle with high-dimensional, multi-modal posterior distributions, frequently becoming trapped in local optima. The EMJMCMC R package addresses these challenges by implementing advanced Evolutionary and Mode Jumping MCMC algorithms designed to explore complex model spaces efficiently.

Methodology: MJMCMC and GMJMCMC

The core of the package lies in two sophisticated algorithms developed to overcome the limitations of standard samplers:

  1. Mode Jumping MCMC (MJMCMC): This algorithm utilizes large, non-local jumps to traverse regions of low probability, allowing the sampler to move between distinct high-probability modes in the posterior distribution. This "brave" sampling strategy is essential for escaping local optima in rugged probability landscapes.
  2. Genetically Modified MJMCMC (GMJMCMC): This approach integrates evolutionary logic with mode jumping. By treating model features as "genes," the algorithm employs crossover and mutation operators to discover complex non-linear transformations and logical combinations of variables.

A standout feature of EMJMCMC is its automated feature engineering. Unlike standard variable selection, which merely chooses from existing columns, the evolutionary component allows the algorithm to evolve the model structure itself. It can generate new features using a library of mathematical transformations (e.g., sin, sigmoid, tanh) and logical operators (e.g., AND, NOT), effectively discovering optimal predictors like sigmoid(X1 * X2) that might be missed by traditional methods.

Implementation and Flexible Estimators

The toolbox is designed with a modular architecture, allowing users to plug in different estimators depending on their data type. This flexibility is a core strength of the package. For example, the estimate.bas.glm function enables fine-tuned Bayesian estimation for generalized linear models using a variety of distribution families and priors, such as AIC, BIC, or IC priors.

Parameters for the BAS GLM estimator, demonstrating support for diverse distribution families and Bayesian priors.

To handle large-scale datasets that exceed memory capacity, the package includes the estimate.bigm function. This estimator supports chunked processing, ensuring computational feasibility for massive data frames by staying within memory limits.

Parameters for the bigm estimator, highlighting the chunksize parameter for efficient large-scale data processing.

Furthermore, EMJMCMC supports parallel processing to run multiple MCMC chains simultaneously, which is crucial for the heavy lifting required by BMA in high-dimensional settings.

Applications and Results

The package's utility is demonstrated through both synthetic and real-world datasets. Simulated data examples show the algorithm's ability to recover ground-truth non-linear relationships. In a real-world context, the package includes breast cancer datasets to showcase logic regression and predictive performance in medical research.

Key functions such as runemjmcmc provide essential outputs for researchers, including:

  • p.post: Posterior probabilities for each covariate or evolved feature.
  • post.populi: The total mass of the visited model space, offering a metric for the thoroughness of the exploration.

Conclusion and Limitations

While EMJMCMC offers a powerful framework for complex Bayesian modeling, it has certain limitations. Currently, it does not support latent random fields (e.g., for integration with INLA), and some features like the estimate.elnet function for elastic networks are still in development. Additionally, the evolutionary component introduces several hyperparameters (mutation rates, crossover probabilities, etc.) that may require expert tuning to optimize performance. Despite these factors, EMJMCMC remains a robust "expert toolbox" for researchers tackling high-dimensional model selection challenges.

Summary updated

Abstract

05316>, which allow for estimating posterior model probabilities and Bayesian model averaging across a wide set of Bayesian models including linear, generalized linear, generalized linear mixed, generalized nonlinear, generalized nonlinear mixed, and logic regression models.
BibTeX
@inproceedings{hubin2024emjmcmc,
  author = {Aliaksandr Hubin and Waldir Leoncio},
  title = {{EMJMCMC: Evolutionary Mode Jumping Markov Chain Monte Carlo Expert Toolbox}},
  booktitle = {Openalex Unknown},
  year = {2024},
  doi = {10.32614/cran.package.emjmcmc},
}
Figure