Measuring the effective complexity of cosmological models

M. Kunz, Roberto Trotta, David ParkinsonPhysical 2006 · DOI

TL;DR

This paper introduces "Bayesian complexity," a new statistical tool to quantify how many parameters a dataset can actually constrain, going beyond just the number of parameters in a model. They apply this to cosmic microwave background data, showing it helps determine the true complexity needed to describe the universe and revealing that current data can constrain up to seven parameters, though six are sufficient for the standard cosmological model.

What Problem Does This Paper Address?

Cosmologists are constantly trying to build the simplest possible model that accurately describes the universe. When comparing different models, a key challenge is balancing model fit with model complexity. A more complex model (with more parameters) can often fit the data better, but this can lead to "overfitting," where the model captures noise rather than true underlying patterns. Traditional Bayesian model selection uses the "evidence" (or marginal likelihood) to penalize complexity, but it doesn't explicitly tell us how many parameters the data can actually constrain. This paper addresses this gap by developing a quantitative measure of "effective complexity" that complements the model evidence, helping to distinguish between models where extra parameters are genuinely supported by the data versus those where they are not.

Methods

The core contribution is the introduction of the Bayesian complexity ($C_b$). This measure is derived from the Kullback-Leibler (KL) divergence between the prior and posterior probability distributions of a model's parameters. Essentially, it quantifies the information gained about the parameters from the data. A high Bayesian complexity indicates that many parameters have been well-constrained.

The authors derive $C_b$ for linear models, showing it relates to the trace of the posterior covariance matrix. They also discuss its relationship to the model likelihood (evidence), which is related to the determinant of the Fisher information matrix. The key idea is that while the likelihood tells you how well a model fits the data, the Bayesian complexity tells you how well the data has constrained the model's parameters. They propose using these two measures together for model selection: a model with a high likelihood and a Bayesian complexity that matches the number of parameters it actually needs is preferred.

Datasets

The paper uses publicly available cosmological data, specifically:

  • Cosmic Microwave Background (CMB) anisotropy measurements from WMAP (first-year and three-year data), ACBAR, CBI, VSA, and Boomerang 2003.
  • Hubble Space Telescope (HST) measurements of the Hubble parameter ($H_0 = 72 \pm 8$ km/s/Mpc).

These datasets are used to test various cosmological models, including the standard $\Lambda$CDM model with different extensions (e.g., adding curvature, a scalar spectral index $n_s$, and reionization optical depth $\tau$).

Key Insights and Ideas

  • Beyond Input Complexity: The number of parameters in a model ($C_0$) is a crude measure. Bayesian complexity ($C_b$) provides a more nuanced view of how many parameters the data can actually constrain.
  • Complementary to Evidence: Bayesian complexity is not a replacement for model evidence but a crucial complement. It helps resolve ambiguities where models have similar evidence but different numbers of parameters.
  • Data-Driven Parameter Count: The paper demonstrates that $C_b$ can reveal the "effective" number of parameters that are well-measured by the data, even if the model has more "input" parameters.
  • CMB Data Power: The analysis shows that the WMAP 3-year data, combined with $H_0$ measurements, significantly improves the constraints on cosmological parameters, allowing for the simultaneous measurement of parameters like $n_s$ and $\tau$.
  • Curvature is Disfavored: The study strongly disfavors a non-zero spatial curvature for the universe, consistent with previous findings.

Key Results

  • For a toy linear model, they show that Bayesian complexity correctly identifies the true number of parameters used to generate the data, even when the model likelihood might suggest otherwise due to noise or limited data points.
  • Applying the method to CMB and HST data, they find that the data can constrain at least seven effective parameters within the considered $\Lambda$CDM extensions.
  • However, they conclude that a model with six parameters (specifically, the base $\Lambda$CDM parameters plus $n_s$ and $\tau$) is sufficient to explain the current data, with spatial curvature being disfavored.
  • The WMAP 3-year data shows a significant improvement in constraining power compared to WMAP 1-year data, particularly for the $n_s$ and $\tau$ parameters.

Limitations and Future Work

  • Gaussian Assumption: The derivation and application of Bayesian complexity are most straightforward for Gaussian posterior distributions. The paper acknowledges that for strongly non-Gaussian posteriors (e.g., those involving curvature), the standard point estimators for the KL divergence might not be representative. They suggest using a modified estimator (average of $\chi^2$ at mean and mode) in such cases but note that further research is needed for optimal estimators in non-Gaussian scenarios.
  • Prior Dependence: While Bayesian complexity aims to quantify data constraints, the interpretation of "well-measured" still implicitly depends on the prior. The paper notes that for parameters with very narrow priors, even if the data doesn't strongly constrain them, they might appear "measured" in a relative sense.
  • Future Work: The authors suggest that Bayesian complexity is a valuable tool for routine use in future data analyses, particularly for model building and assessment. They highlight the need for further study of its performance in situations with strongly non-Gaussian posteriors.

Summary updated

Abstract

We introduce a statistical measure of the effective model complexity, called the Bayesian complexity. We demonstrate that the Bayesian complexity can be used to assess how many effective parameters a set of data can support and that it is a useful complement to the model likelihood (the evidence) in model selection questions. We apply this approach to recent measurements of cosmic microwave background anisotropies combined with the Hubble Space Telescope measurement of the Hubble parameter. Using mildly noninformative priors, we show how the 3-year WMAP data improves on the first-year data by being able to measure both the spectral index and the reionization epoch at the same time. We also find that a nonzero curvature is strongly disfavored. We conclude that although current data could constrain at least seven effective parameters, only six of them are required in a scheme based on the $\ensuremath{\Lambda}\mathrm{CDM}$ concordance cosmology.
BibTeX
@inproceedings{kunz2006measuring,
  author = {M. Kunz and Roberto Trotta and David Parkinson},
  title = {{Measuring the effective complexity of cosmological models}},
  booktitle = {Physical},
  year = {2006},
  doi = {10.1103/physrevd.74.023503},
}
Figure