Prior Sample Size Extensions for Assessing Prior Impact and Prior-Likelihood Discordance

Matthew Reimherr, Xiao‐Li Meng, Dan L. Nicolae

Abstract This paper outlines a framework for quantifying the prior’s contribution to posterior inference in the presence of prior-likelihood discordance, a broader concept than the usual notion of prior-likelihood conflict. We achieve this dual purpose by extending the classic notion of prior sample size, M, in three directions: (I) estimating M beyond conjugate families; (II) formulating M as a relative notion that is as a function of the likelihood sample size k, M(k), which also leads naturally to a graphical diagnosis; and (III) permitting negative M, as a measure of prior-likelihood conflict, that is, harmful discordance. Our asymptotic regime permits the prior sample size to grow with the likelihood data size, hence making asymptotic arguments meaningful for investigating the impact of the prior relative to that of likelihood. It leads to a simple asymptotic formula for quantifying the impact of a proper prior that only involves computing a centrality and a spread measure of the prior and the posterior. We use simulated and real data to illustrate the potential of the proposed framework, including quantifying how weak is a ‘weakly informative’ prior adopted in a study of lupus nephritis. Whereas we take a pragmatic perspective in assessing the impact of a prior on a given inference problem under a specific evaluative metric, we also touch upon conceptual and theoretical issues such as using improper priors and permitting priors with asymptotically non-vanishing influence.

Introduction

In Bayesian statistics, the selection of a prior distribution is often the most critical and controversial step. While researchers frequently employ "weakly informative" priors to regularize models, there has historically been no standard, data-driven method to quantify exactly how much influence these priors exert relative to the observed data. This paper introduces a flexible framework to quantify the Prior Sample Size (PSS), effectively translating the information contained in a prior into an equivalent number of data points.

Quantifying Prior Influence and Discordance

The authors define PSS by matching the expected posterior loss (typically Mean Squared Error) between two scenarios:

  1. A posterior calculated using the target prior and the observed data.
  2. A posterior calculated using a non-informative "baseline" prior but with a different amount of data.

The "extra" data required in the second scenario to match the error of the first defines the PSS ($M$). This approach extends traditional PSS in several key ways:

  • Relative PSS ($M(k)$): Rather than a static value, PSS is treated as a function of the likelihood sample size ($k$). This allows for a graphical diagnostic of the relationship between the prior and the data.
  • Negative PSS: If a prior is so far from the truth that it increases the posterior error compared to using no prior at all, the framework assigns a negative $M$ value, providing a formal measure of prior-likelihood conflict.
  • Asymptotic Formula: The authors derive a "rule-of-thumb" formula that requires only the mean and variance (centrality and spread) of the prior and the posterior, making it accessible for quick assessments.
  • Finite-Sample Bootstrap: For complex, non-conjugate models where analytical formulas are unavailable, a subsampling/bootstrap approach is proposed to estimate $M(k)$ numerically.

Key Insights: Discordance vs. Conflict

A central theme of the research is the distinction between "discordance" and "conflict." The authors argue that discordance is a broad concept describing any lack of harmony between the prior and the likelihood. This includes "lucky" priors—where the prior mean is very close to the true parameter, potentially providing up to twice as much information as its nominal size (the "Super-Information" phenomenon). Conversely, "conflict" is reserved for cases where the prior is actively harmful ($M < 0$).

Theoretical analysis also reveals a "-1 slope limit." The authors prove that the slope of $M(k)$ is bounded below by -1. A slope of -1 indicates a prior so conflicting that every new data point added to the likelihood is essentially neutralized by the prior's misinformation.

Empirical Application: Lupus Nephritis

The framework was validated using a real-world dataset of 55 patients to predict "stage V" lupus based on immunoglobulin levels (IgG3 and IgG4). Researchers often use a Cauchy prior (scale 2.5) as a "weakly informative" default in such studies. By applying the PSS framework, the authors demonstrated that this prior was equivalent to adding only 2 to 6 data points. In the context of a 55-patient study, this represents less than 10% of the total information, providing empirical confirmation that the prior is indeed "weak."

Conclusion and Limitations

This framework provides a pragmatic tool for assessing the impact of a prior on a specific inference problem. While the bootstrap method can be computationally demanding for high-dimensional models, the asymptotic formulas offer a reliable and efficient alternative. Future work aims to extend these definitions to non-IID data, such as time-series or spatial models, where the definition of "sample size" is more complex. Despite the sensitivity of MSE to reparameterization, the authors maintain that it remains the most practical metric for most researchers to understand prior impact.

Summary updated

Abstract

Abstract This paper outlines a framework for quantifying the prior’s contribution to posterior inference in the presence of prior-likelihood discordance, a broader concept than the usual notion of prior-likelihood conflict. We achieve this dual purpose by extending the classic notion of prior sample size, M, in three directions: (I) estimating M beyond conjugate families; (II) formulating M as a relative notion that is as a function of the likelihood sample size k, M(k), which also leads naturally to a graphical diagnosis; and (III) permitting negative M, as a measure of prior-likelihood conflict, that is, harmful discordance. Our asymptotic regime permits the prior sample size to grow with the likelihood data size, hence making asymptotic arguments meaningful for investigating the impact of the prior relative to that of likelihood. It leads to a simple asymptotic formula for quantifying the impact of a proper prior that only involves computing a centrality and a spread measure of the prior and the posterior. We use simulated and real data to illustrate the potential of the proposed framework, including quantifying how weak is a ‘weakly informative’ prior adopted in a study of lupus nephritis. Whereas we take a pragmatic perspective in assessing the impact of a prior on a given inference problem under a specific evaluative metric, we also touch upon conceptual and theoretical issues such as using improper priors and permitting priors with asymptotically non-vanishing influence.
BibTeX
@article{reimherr2021prior,
  author = {Matthew Reimherr and Xiao‐Li Meng and Dan L. Nicolae},
  title = {{Prior Sample Size Extensions for Assessing Prior Impact and Prior-Likelihood Discordance}},
  journal = {Journal of the Royal Statistical Society Series B (Statistical Methodology)},
  year = {2021},
  doi = {10.1111/rssb.12414},
}
Figure