Asymptotics for power posterior mean estimation

Marco Avella Medina, Ruchira Ray, Marco Avella-Medina, Cynthia RushAllerton 2023 · DOI

TL;DR

This paper investigates "power posteriors," a Bayesian inference technique that offers robustness to model misspecification by downweighting the influence of the likelihood. The authors prove that the moments of these power posteriors converge to those of a standard Bernstein-von Mises distribution, and crucially, that the power posterior mean is asymptotically equivalent to the Maximum Likelihood Estimator (MLE). This means that while power posteriors offer robustness, they don't sacrifice the desirable asymptotic properties of the MLE for point estimation.

What Problem Does This Paper Address?

In Bayesian statistics, the posterior distribution is central to inference. It's calculated by combining a prior belief about parameters with the evidence from data (the likelihood). However, if the assumed statistical model is incorrect (misspecified), the standard posterior can be unreliable. "Power posteriors" (or $\alpha$-posteriors) offer a solution by raising the likelihood to a fractional power ($\alpha < 1$). This effectively "tempers" the likelihood, reducing its influence and making the posterior more robust to model errors.

While previous work showed that power posteriors still exhibit asymptotic normality (the Bernstein-von Mises property) even under misspecification, it was unclear how this tempering affected specific point estimators derived from these posteriors, particularly the posterior mean. This paper aims to bridge that gap by rigorously analyzing the asymptotic behavior of the moments of the power posterior and, consequently, the power posterior mean estimator.

Methods

The authors build upon existing proofs for Bernstein-von Mises theorems for power posteriors. Their approach involves:

  • Moment Convergence: They extend previous results to show that the $k$-th moment of the power posterior converges in probability to the $k$-th moment of the limiting normal distribution predicted by the Bernstein-von Mises theorem. This is achieved by adapting techniques that relate the "closeness" of likelihood ratios to the "closeness" of moments, rather than total variation distance.
  • Asymptotic Equivalence to MLE: Using the established moment convergence, they then prove that the power posterior mean estimator is asymptotically equivalent to the Maximum Likelihood Estimator (MLE). This means that as the sample size grows large, the power posterior mean behaves in the same way as the MLE.

The proofs rely on several technical assumptions about the statistical model, the prior distribution, and the behavior of the likelihood function under misspecification. Notably, their assumptions are slightly stronger than some previous work but still allow for model misspecification and non-i.i.d. data.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Robustness Without Sacrificing Asymptotic Efficiency: The most significant insight is that tempering the likelihood for robustness does not compromise the asymptotic efficiency of the posterior mean. It behaves like the MLE in the limit, which is a desirable property.
  • Power of Moment Analysis: The paper demonstrates the utility of analyzing moment convergence to understand the asymptotic behavior of estimators derived from tempered posteriors.
  • Generalizability of Robustness: The findings suggest that this phenomenon of asymptotic equivalence to the MLE might extend to other point estimators derived from power posteriors, such as the posterior median or mode.

Key Results

  • Theorem 1: For any integer $k$ up to a certain bound ($k_0$), the $k$-th moment of the power posterior converges in probability to the $k$-th moment of the limiting normal distribution.
  • Theorem 2: The power posterior mean estimator has the same asymptotic distribution as the Maximum Likelihood Estimator (MLE). This implies that the tempering parameter $\alpha$ does not affect the first-order asymptotic properties of the posterior mean.

Limitations and Future Work

  • Stronger Assumptions: The paper acknowledges that their assumption (A3) for moment convergence is slightly stronger than the posterior concentration assumption used in some prior Bernstein-von Mises results. Future work could aim to relax this.
  • Data-Driven $\alpha$: The current work assumes a fixed, constant tempering parameter $\alpha$. An important future direction is to establish similar guarantees for power posteriors where $\alpha$ is selected in a data-driven way (i.e., $\alpha$ can change with the sample size $n$).
  • Other Estimators: The authors suggest investigating whether this asymptotic equivalence to the MLE holds for other point estimators like the posterior median and mode.

Summary updated

BibTeX
@inproceedings{medina2023asymptotics,
  author = {Marco Avella Medina and Ruchira Ray and Marco Avella-Medina and Cynthia Rush},
  title = {{Asymptotics for power posterior mean estimation}},
  booktitle = {Allerton},
  year = {2023},
  doi = {10.1109/allerton58177.2023.10313460},
}
Figure