Abstract
We consider approximate Bayesian model choice for model selection problems that involve models whose Fisher-information matrices may fail to be invertible along other competing submodels. Such singular models do not obey the regularity conditions underlying the derivation of Schwarz's Bayesian information criterion (BIC) and the penalty structure in BIC generally does not reflect the frequentist large-sample behavior of their marginal likelihood. While large-sample theory for the marginal likelihood of singular models has been developed recently, the resulting approximations depend on the true parameter value and lead to a paradox of circular reasoning. Guided by examples such as determining the number of components of mixture models, the number of factors in latent factor models or the rank in reduced-rank regression, we propose a resolution to this paradox and give a practical extension of BIC for singular model selection problems.
Introduction: The Failure of Standard BIC
Model selection is a fundamental task in statistics, often involving the choice of the "best" model from a set of candidates—such as determining the number of clusters in a dataset or the rank of a regression matrix. The Bayesian Information Criterion (BIC) has long been the standard tool for this purpose, balancing model fit against complexity using a penalty based on the number of parameters. However, BIC's derivation relies on the assumption of "regularity," where the Fisher information matrix is invertible.
Many modern statistical models, including mixture models, hidden Markov models, and neural networks, are "singular." In these cases, the standard BIC penalty ($d/2 \log n$) is often too harsh, leading to the under-selection of complex models. While Singular Learning Theory (SLT) provides more accurate asymptotic expansions for the marginal likelihood, these expansions typically depend on the unknown true parameter value, creating a "circular reasoning" paradox where the correct penalty depends on the very truth we are trying to uncover.
The sBIC Framework: Resolving the Paradox
To address this, we introduce the singular BIC (sBIC). This criterion leverages the geometric complexity of a model's parameter space near singularities, characterized by "learning coefficients" ($\lambda$). Instead of a constant penalty based solely on dimension, sBIC employs a data-dependent weighting system across a partially ordered set (poset) of candidate models.
By organizing models into a hierarchy (e.g., a 2-component mixture as a submodel of a 3-component mixture), sBIC calculates a proxy for the marginal likelihood that effectively "learns" the appropriate penalty from the data itself. This adaptive weighting allows sBIC to incorporate prior-dependent bounds and geometric complexity without requiring the true parameter value a priori.
Empirical Performance and Results
The performance of sBIC was evaluated through extensive simulations and real-world applications, including reduced-rank regression and factor analysis. A key finding is that sBIC provides a much closer approximation to the true marginal likelihood than standard BIC, especially as sample size $n$ increases.
In simulations of reduced-rank regression, sBIC identified the true rank at sample sizes as small as $n=300$, whereas standard BIC required significantly more data to overcome its inherent over-penalization. Furthermore, in applications to exchange rate data, sBIC's posterior model probabilities closely mirrored those obtained through computationally intensive Markov Chain Monte Carlo (MCMC) methods. This provides "gold standard" accuracy at the speed of a simple maximum likelihood calculation, avoiding the massive computational overhead of full Bayesian simulations.
Conclusion
The sBIC offers a theoretically grounded and computationally efficient alternative to BIC for singular model selection. By incorporating learning coefficients and a poset-based weighting scheme, it resolves the paradoxes of singular learning theory and provides a consistent selection criterion that aligns with full Bayesian inference. While deriving learning coefficients for new, exotic architectures remains a mathematical challenge, sBIC represents a significant step forward in making advanced singular learning theory accessible for practical statistical modeling.
Summary updated