Minimum Message Length Ridge Regression for Generalized Linear Models

Daniel F. Schmidt, Enes MakalicAustralasian Conference on Artificial Intelligence 2013 · DOI

TL;DR

This paper introduces a novel Minimum Message Length (MML) based criterion for selecting features and estimating parameters in Generalized Linear Models (GLMs). This unified approach offers a principled way to perform both model selection and parameter estimation simultaneously, outperforming traditional methods like AICc in various simulations and real-world datasets.

What Problem Does This Paper Address?

Traditional regression models often assume normally distributed target variables. Generalized Linear Models (GLMs) extend this by allowing for various distributions (e.g., binomial, Poisson, gamma) and using a "link function" to connect the linear predictor to the mean of the target variable. However, selecting the right set of predictor variables (features) and estimating the model's parameters (coefficients and regularization strength) are often treated as separate, sequential problems. This can lead to suboptimal models. Furthermore, standard regularization techniques like ridge regression, while effective at handling multicollinearity, often rely on information criteria like AIC or cross-validation for hyperparameter tuning, which can be computationally intensive or less principled. This paper aims to provide a single, information-theoretic framework that addresses feature selection, parameter estimation, and regularization hyperparameter tuning for GLMs in a unified manner.

Methods

The core of this paper is the application of the Minimum Message Length (MML) principle to Generalized Linear Models (GLMs) with ridge regression. MML is an information-theoretic approach to inductive inference that seeks the model that most compresses the data, thereby balancing model complexity with goodness of fit.

The authors adapt the MML87 approximation, which provides a computationally tractable way to calculate message length. They extend the standard GLM framework by:

  • Unified Framework: Treating parameter estimation (regression coefficients $\alpha, \beta$), dispersion parameter $\phi$, and the ridge regularization hyperparameter $\lambda$ as parameters to be estimated within the MML framework.
  • Bayesian Interpretation of Ridge Regression: Leveraging the Bayesian interpretation of ridge regression, where the regularization term arises from a multivariate normal prior on the regression coefficients. This prior is parameterized by $\lambda$.
  • Corrected Fisher Information: To handle the regularization hyperparameter $\lambda$ within the MML framework, they employ a "corrected" Fisher information matrix. This addresses potential issues when the prior becomes very concentrated around the origin (i.e., very large $\lambda$).
  • Feature Selection: The MML principle is extended to select relevant covariates by treating the choice of a subset of covariates as a model selection problem. The message length is minimized over all possible subsets of covariates.

The paper outlines specific MML formulas for different GLM scenarios, including cases with and without covariates, and for various target distributions. They also propose using posterior mode (MAP) estimates as a computationally efficient surrogate for exact MML estimates, especially for larger sample sizes.

Datasets

The paper evaluates their proposed MML GLM ridge criterion on:

  • Simulated Data: Experiments were conducted using simulated data for normal, binomial, and Poisson distributions. The simulations involved varying sample sizes ($n \in \{25, 50, 100, 250\}$) and correlation structures of the design matrix (controlled by $\rho \in \{0.1, 0.5, 0.9\}$).
  • Real-world Datasets: Three real-world datasets were used for model selection experiments:
  • Pima Indians dataset (binary targets, $q=8$ covariates, $n=768$)
    • Diabetes dataset (normal targets, $q=10$ covariates, $n=442$)
    • Boston Housing dataset (normal targets, $q=14$ covariates, $n=506$) These datasets were augmented with four additional noise covariates to make the model selection task more challenging.

Key Insights and Ideas

  • Unified Inference: The most significant insight is the development of a single, information-theoretic criterion (MML) that can simultaneously perform feature selection, parameter estimation, and regularization hyperparameter tuning for GLMs. This contrasts with traditional approaches that often treat these tasks separately.
  • Principled Regularization: Using MML for ridge regression provides a principled way to estimate the regularization strength ($\lambda$), avoiding reliance on less theoretically grounded methods like cross-validation or AIC.
  • Model Complexity Balancing: The MML principle inherently balances model complexity against goodness of fit, acting as a natural safeguard against overfitting, which is a crucial aspect of robust statistical modeling.
  • Broad Applicability: The proposed method is shown to be highly general, accommodating a wide range of target distributions commonly used in GLMs.

Key Results

  • Parameter Estimation: In parameter estimation simulations, the MML estimates were found to be competitive with, or superior to, the corrected Akaike Information Criterion (AICc) estimates. For normal regression models, MML was superior in all but one case. The improvements were generally larger for higher levels of correlation in the design matrix.
  • Model Selection: On real-world datasets, the MML criterion was competitive with, or superior to, AICc for model selection across various sample sizes and dataset types. The performance difference was particularly noticeable for the Pima Indians dataset.
  • Efficiency: While exact MML estimation can be computationally intensive, the paper proposes using posterior mode (MAP) estimates as an efficient surrogate that performs well, especially for moderate to large sample sizes.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that calculating exact MML estimates can be computationally expensive, particularly for complex models or large datasets. The use of MAP estimates is presented as a practical solution to mitigate this.
  • Approximations: The MML87 approximation is used, which is generally accurate but still an approximation. The "corrected" Fisher information matrix is also an adaptation to handle the hyperparameter.
  • Hyperparameter Coding: The paper mentions that the codelength for the hyperparameter $\lambda$ is approximated using an asymptotic formula, which might not be optimal but is expected to have a limited impact on the overall inference for a single hyperparameter.
  • Future Work: The paper's success in unifying parameter estimation and model selection suggests potential for further exploration in more complex modeling scenarios and with different types of regularization. The broad applicability to various GLMs also opens doors for applying this framework to a wider array of statistical problems.

Summary updated

BibTeX
@inproceedings{schmidt2013minimum,
  author = {Daniel F. Schmidt and Enes Makalic},
  title = {{Minimum Message Length Ridge Regression for Generalized Linear Models}},
  booktitle = {Australasian Conference on Artificial Intelligence},
  year = {2013},
  doi = {10.1007/978-3-319-03680-9\_41},
}
Figure