TL;DR
This paper tackles the challenge of reliably assessing the performance of importance sampling (IS) estimators for Bayesian posterior expectations, especially when ground truth is unavailable. The authors propose a new diagnostic based on "tail dependence" between the numerator and denominator weights of the IS estimator, arguing it captures crucial information missed by existing methods.
What Problem Does This Paper Address?
Estimating posterior expectations is fundamental in Bayesian inference, but evaluating the quality of these estimates is tricky without knowing the true value. Existing methods like the "k-hat" diagnostic for self-normalized importance sampling (SNIS) can be misleading because they treat the numerator and denominator of the SNIS estimator separately. However, the SNIS estimator is a ratio, and the dependence between the numerator and denominator estimates plays a critical role in its overall accuracy. This paper aims to develop a diagnostic that accounts for this crucial inter-dependence, leading to more reliable performance assessments.
Methods
The core of the paper's contribution lies in proposing a new diagnostic that captures the tail dependence between the weights used for the numerator and denominator in the SNIS estimator.
- Existing Diagnostic (k-hat): The paper discusses the "k-hat" diagnostic, which estimates a parameter (k) from a generalized Pareto distribution (GPD) fitted to the importance sampling weights. This parameter is used to infer the reliability of the estimator. However, applying this separately to numerator and denominator weights is shown to be insufficient.
- Proposed Diagnostic (Tail Dependence): The authors introduce a diagnostic based on the concept of upper tail dependence coefficient ($\lambda_U$). This coefficient measures the probability that one variable is large given that another variable is also large. They hypothesize that strong tail dependence between the numerator and denominator weights can lead to error cancellation in the ratio, thus improving the estimator's performance. They explore semi-parametric estimators for this tail dependence, drawing from techniques used in financial mathematics.
- Generalized SNIS: To better understand and test their ideas, the paper also discusses a generalized SNIS estimator that allows for couplings between the numerator and denominator sampling processes, going beyond the standard independent sampling.
Datasets
Not discussed in the paper. The experiments are conducted on a Bayesian linear regression model where the exact posterior expectation can be computed, serving as a synthetic ground truth.
Key Insights and Ideas
- SNIS is a Ratio, Not Just Two Estimates: The paper emphasizes that the SNIS estimator's reliability is not solely determined by the quality of the numerator and denominator estimates in isolation, but crucially by their joint behavior due to the ratio structure.
- Tail Dependence as a Key Indicator: The authors propose that the way extreme weights in the numerator and denominator co-occur (tail dependence) is a strong indicator of estimator performance. High tail dependence can lead to beneficial error cancellation.
- Limitations of Separate Diagnostics: The paper demonstrates that simply looking at diagnostics like k-hat for the numerator and denominator separately can miss important information about their interaction, leading to potentially unreliable assessments.
- Beyond Pearson's Correlation: The paper implicitly argues for the need for dependence measures beyond simple correlation (like Pearson's $\rho$), as tail dependence captures a specific and relevant form of dependency for ratio estimators.
Key Results
- Empirical Validation: Through experiments on Bayesian linear regression, the authors show that their proposed tail dependence diagnostic, when combined with k-hat diagnostics, provides a more accurate assessment of estimator performance than k-hat alone.
- Explaining Performance Gaps: In scenarios where standard k-hat diagnostics are similar for competing methods, the tail dependence metric helps explain remaining differences in performance, highlighting its added value.
- High-Dimensional Challenges: The paper notes that in high-dimensional settings, all methods can struggle, but the proposed diagnostic still offers insights into relative performance.
Limitations and Future Work
- Computational Cost: While not explicitly detailed, estimating tail dependence can be computationally more intensive than simpler diagnostics.
- Choice of Copula: The paper mentions using a Gumbel copula for estimating tail dependence and notes it performed better than a t-copula, suggesting that the choice of copula model might influence the diagnostic's effectiveness.
- Future Work: The authors plan to test further tail dependence metrics and apply their diagnostic to a wider range of Bayesian models. They also aim to explore more sophisticated generalized SNIS estimators.
Summary updated