TL;DR
This paper investigates the "variational gap" in variational inference, a common approximation technique. The authors show that this gap is directly related to the variance of likelihood ratios, suggesting that reducing this variance through methods like averaging can lead to more accurate approximations. This provides a theoretical justification for existing variance reduction techniques in VI.
What Problem Does This Paper Address?
Variational Inference (VI) is a powerful technique for approximating intractable posterior distributions in probabilistic models. A core challenge in VI is that it often relies on maximizing a lower bound on the marginal likelihood, known as the Evidence Lower Bound (ELBO). The difference between the true marginal likelihood and the ELBO is called the "variational gap." This gap represents a bias in the VI approximation. While methods like importance weighting (IW) can reduce this gap by using multiple samples, understanding why and how much they reduce it, and how this relates to the variance of the samples, has been an ongoing area of research. This paper aims to provide a clearer theoretical understanding of this relationship, specifically connecting the variational gap to the dispersion (variance) of likelihood ratios.
Methods
The paper's core contribution lies in its theoretical analysis. The authors introduce the concept of the "variational gap" and relate it to the dispersion of the likelihood ratio, $X = p(v,h)/q(h)$. They leverage Jensen's inequality and properties of concave functions (specifically, the logarithm) to derive bounds on this gap.
Key techniques include:
- Importance Weighted Lower Bound (IWLB): The paper discusses a family of lower bounds derived from importance sampling, where multiple samples of the latent variable $h$ are drawn from a proposal distribution $q(h)$ and used to estimate the marginal likelihood.
- Central Tendency and Dispersion: The analysis focuses on the relationship between the mean ($\mu_X$) and median ($\nu_X$) of the likelihood ratio $X$, and their respective log-transformed counterparts. They use measures of dispersion like the standard deviation ($\sigma_X$, $\sigma_Y$) to bound the gap.
- Taylor Approximation: For bounding the gap, a first-order Taylor approximation of the logarithm function is used to establish a linear upper bound.
Datasets
Not discussed in the paper. The paper focuses on theoretical analysis rather than empirical evaluation on specific datasets.
Key Insights and Ideas
- Variational Gap as a Measure of Dispersion: The paper's central insight is that the variational gap is not just an arbitrary bias but is directly linked to the dispersion (variance) of the likelihood ratio $X$. This means that if the likelihood ratios are tightly clustered around their mean, the variational gap will be smaller.
- Variance Reduction Justified: This theoretical connection provides a strong justification for using variance reduction techniques in VI. Methods that aim to make the distribution of likelihood ratios more concentrated (e.g., through averaging or antithetic sampling) are shown to directly reduce the variational gap.
- Median vs. Mean: The analysis highlights the importance of considering both the mean and median of the likelihood ratio and their log-transformed versions when bounding the variational gap.
- Non-asymptotic Bounds: The paper aims to provide non-asymptotic bounds, meaning the results hold for any number of samples $K$, not just in the limit of infinite samples.
Key Results
The paper derives theoretical bounds on the variational gap. Specifically, Proposition 2 establishes that for a positive random variable $X$ and its logarithm $Y = \log X$, under certain conditions on their medians and means, the gap $\log \mu_X - \mu_Y$ can be bounded by terms involving the dispersion of $X$ and $Y$. Corollary 3 summarizes this by showing that the gap $\log p(v) - E[\log X]$ is bounded by a combination of the standard deviation of $X$ and $Y$, scaled by terms related to the mean of $X$.
Limitations and Future Work
- Assumptions: The theoretical results rely on certain assumptions, such as the random variable $X$ being positive and belonging to $L_p$ for some $p > 1$. The applicability of these bounds in practice might depend on how well these assumptions are met.
- Practical Implementation: While the paper provides theoretical justification, it doesn't detail specific algorithms or empirical studies to demonstrate the practical benefits of these bounds. Future work could involve implementing and evaluating variance reduction techniques based on these theoretical insights.
- Further Exploration of Dispersion Measures: The paper focuses on standard deviation as a measure of dispersion. Future work could explore other dispersion measures and their impact on the variational gap.
Summary updated