TL;DR
This paper introduces "Bayesian complexity," a new statistical tool to quantify how many parameters a dataset can actually constrain, going beyond just the number of parameters in a model. They apply this to cosmic microwave background data, showing it helps determine the true complexity needed to describe the universe and revealing that current data can constrain up to seven parameters, though six are sufficient for the standard cosmological model.
What Problem Does This Paper Address?
Cosmologists are constantly trying to build the simplest possible model that accurately describes the universe. When comparing different models, a key challenge is balancing model fit with model complexity. A more complex model (with more parameters) can often fit the data better, but this can lead to "overfitting," where the model captures noise rather than true underlying patterns. Traditional Bayesian model selection uses the "evidence" (or marginal likelihood) to penalize complexity, but it doesn't explicitly tell us how many parameters the data can actually constrain. This paper addresses this gap by developing a quantitative measure of "effective complexity" that complements the model evidence, helping to distinguish between models where extra parameters are genuinely supported by the data versus those where they are not.
Methods
The core contribution is the introduction of the Bayesian complexity ($C_b$). This measure is derived from the Kullback-Leibler (KL) divergence between the prior and posterior probability distributions of a model's parameters. Essentially, it quantifies the information gained about the parameters from the data. A high Bayesian complexity indicates that many parameters have been well-constrained.
The authors derive $C_b$ for linear models, showing it relates to the trace of the posterior covariance matrix. They also discuss its relationship to the model likelihood (evidence), which is related to the determinant of the Fisher information matrix. The key idea is that while the likelihood tells you how well a model fits the data, the Bayesian complexity tells you how well the data has constrained the model's parameters. They propose using these two measures together for model selection: a model with a high likelihood and a Bayesian complexity that matches the number of parameters it actually needs is preferred.
Datasets
The paper uses publicly available cosmological data, specifically:
- Cosmic Microwave Background (CMB) anisotropy measurements from WMAP (first-year and three-year data), ACBAR, CBI, VSA, and Boomerang 2003.
- Hubble Space Telescope (HST) measurements of the Hubble parameter ($H_0 = 72 \pm 8$ km/s/Mpc).
These datasets are used to test various cosmological models, including the standard $\Lambda$CDM model with different extensions (e.g., adding curvature, a scalar spectral index $n_s$, and reionization optical depth $\tau$).
Key Insights and Ideas
- Beyond Input Complexity: The number of parameters in a model ($C_0$) is a crude measure. Bayesian complexity ($C_b$) provides a more nuanced view of how many parameters the data can actually constrain.
- Complementary to Evidence: Bayesian complexity is not a replacement for model evidence but a crucial complement. It helps resolve ambiguities where models have similar evidence but different numbers of parameters.
- Data-Driven Parameter Count: The paper demonstrates that $C_b$ can reveal the "effective" number of parameters that are well-measured by the data, even if the model has more "input" parameters.
- CMB Data Power: The analysis shows that the WMAP 3-year data, combined with $H_0$ measurements, significantly improves the constraints on cosmological parameters, allowing for the simultaneous measurement of parameters like $n_s$ and $\tau$.
- Curvature is Disfavored: The study strongly disfavors a non-zero spatial curvature for the universe, consistent with previous findings.
Key Results
- For a toy linear model, they show that Bayesian complexity correctly identifies the true number of parameters used to generate the data, even when the model likelihood might suggest otherwise due to noise or limited data points.
- Applying the method to CMB and HST data, they find that the data can constrain at least seven effective parameters within the considered $\Lambda$CDM extensions.
- However, they conclude that a model with six parameters (specifically, the base $\Lambda$CDM parameters plus $n_s$ and $\tau$) is sufficient to explain the current data, with spatial curvature being disfavored.
- The WMAP 3-year data shows a significant improvement in constraining power compared to WMAP 1-year data, particularly for the $n_s$ and $\tau$ parameters.
Limitations and Future Work
- Gaussian Assumption: The derivation and application of Bayesian complexity are most straightforward for Gaussian posterior distributions. The paper acknowledges that for strongly non-Gaussian posteriors (e.g., those involving curvature), the standard point estimators for the KL divergence might not be representative. They suggest using a modified estimator (average of $\chi^2$ at mean and mode) in such cases but note that further research is needed for optimal estimators in non-Gaussian scenarios.
- Prior Dependence: While Bayesian complexity aims to quantify data constraints, the interpretation of "well-measured" still implicitly depends on the prior. The paper notes that for parameters with very narrow priors, even if the data doesn't strongly constrain them, they might appear "measured" in a relative sense.
- Future Work: The authors suggest that Bayesian complexity is a valuable tool for routine use in future data analyses, particularly for model building and assessment. They highlight the need for further study of its performance in situations with strongly non-Gaussian posteriors.
Summary updated