A Bayesian information criterion for singular models

Mathias Drton, Martyn PlummerJournal of the Royal Statistical Society Series B (Statistical Methodology) 2013

Abstract

We consider approximate Bayesian model choice for model selection problems that involve models whose Fisher-information matrices may fail to be invertible along other competing submodels. Such singular models do not obey the regularity conditions underlying the derivation of Schwarz's Bayesian information criterion (BIC) and the penalty structure in BIC generally does not reflect the frequentist large-sample behavior of their marginal likelihood. While large-sample theory for the marginal likelihood of singular models has been developed recently, the resulting approximations depend on the true parameter value and lead to a paradox of circular reasoning. Guided by examples such as determining the number of components of mixture models, the number of factors in latent factor models or the rank in reduced-rank regression, we propose a resolution to this paradox and give a practical extension of BIC for singular model selection problems.

Introduction: The Failure of Standard BIC

Model selection is a fundamental task in statistics, often involving the choice of the "best" model from a set of candidates—such as determining the number of clusters in a dataset or the rank of a regression matrix. The Bayesian Information Criterion (BIC) has long been the standard tool for this purpose, balancing model fit against complexity using a penalty based on the number of parameters. However, BIC's derivation relies on the assumption of "regularity," where the Fisher information matrix is invertible.

Many modern statistical models, including mixture models, hidden Markov models, and neural networks, are "singular." In these cases, the standard BIC penalty ($d/2 \log n$) is often too harsh, leading to the under-selection of complex models. While Singular Learning Theory (SLT) provides more accurate asymptotic expansions for the marginal likelihood, these expansions typically depend on the unknown true parameter value, creating a "circular reasoning" paradox where the correct penalty depends on the very truth we are trying to uncover.

The sBIC Framework: Resolving the Paradox

To address this, we introduce the singular BIC (sBIC). This criterion leverages the geometric complexity of a model's parameter space near singularities, characterized by "learning coefficients" ($\lambda$). Instead of a constant penalty based solely on dimension, sBIC employs a data-dependent weighting system across a partially ordered set (poset) of candidate models.

Table of learning coefficients ($\lambda$) for specific model hierarchies, illustrating the non-integer penalties required for singular models.

By organizing models into a hierarchy (e.g., a 2-component mixture as a submodel of a 3-component mixture), sBIC calculates a proxy for the marginal likelihood that effectively "learns" the appropriate penalty from the data itself. This adaptive weighting allows sBIC to incorporate prior-dependent bounds and geometric complexity without requiring the true parameter value a priori.

Empirical Performance and Results

The performance of sBIC was evaluated through extensive simulations and real-world applications, including reduced-rank regression and factor analysis. A key finding is that sBIC provides a much closer approximation to the true marginal likelihood than standard BIC, especially as sample size $n$ increases.

Comparison of the log-marginal likelihood ratio approximations (sBIC vs. BIC) against the true values across varying sample sizes $n$.

In simulations of reduced-rank regression, sBIC identified the true rank at sample sizes as small as $n=300$, whereas standard BIC required significantly more data to overcome its inherent over-penalization. Furthermore, in applications to exchange rate data, sBIC's posterior model probabilities closely mirrored those obtained through computationally intensive Markov Chain Monte Carlo (MCMC) methods. This provides "gold standard" accuracy at the speed of a simple maximum likelihood calculation, avoiding the massive computational overhead of full Bayesian simulations.

Conclusion

The sBIC offers a theoretically grounded and computationally efficient alternative to BIC for singular model selection. By incorporating learning coefficients and a poset-based weighting scheme, it resolves the paradoxes of singular learning theory and provides a consistent selection criterion that aligns with full Bayesian inference. While deriving learning coefficients for new, exotic architectures remains a mathematical challenge, sBIC represents a significant step forward in making advanced singular learning theory accessible for practical statistical modeling.

Summary updated

Abstract

We consider approximate Bayesian model choice for model selection problems that involve models whose Fisher-information matrices may fail to be invertible along other competing submodels. Such singular models do not obey the regularity conditions underlying the derivation of Schwarz's Bayesian information criterion (BIC) and the penalty structure in BIC generally does not reflect the frequentist large-sample behavior of their marginal likelihood. While large-sample theory for the marginal likelihood of singular models has been developed recently, the resulting approximations depend on the true parameter value and lead to a paradox of circular reasoning. Guided by examples such as determining the number of components of mixture models, the number of factors in latent factor models or the rank in reduced-rank regression, we propose a resolution to this paradox and give a practical extension of BIC for singular model selection problems.
BibTeX
@article{drton2013bayesian,
  author = {Mathias Drton and Martyn Plummer},
  title = {{A Bayesian information criterion for singular models}},
  journal = {Journal of the Royal Statistical Society Series B (Statistical Methodology)},
  year = {2013},
}
Figure