Abstract
Quantifying the deviation of a probability distribution is challenging when the target distribution is defined by a density with an intractable normalizing constant. The kernel Stein discrepancy (KSD) was proposed to address this problem and has been applied to various tasks including diagnosing approximate MCMC samplers and goodness-of-fit testing for unnormalized statistical models. This article investigates a convergence control property of the diffusion kernel Stein discrepancy (DKSD), an instance of the KSD proposed by Barp et al. (2019). We extend the result of Gorham and Mackey (2017), which showed that the KSD controls the bounded-Lipschitz metric, to functions of polynomial growth. Specifically, we prove that the DKSD controls the integral probability metric defined by a class of pseudo-Lipschitz functions, a polynomial generalization of Lipschitz functions. We also provide practical sufficient conditions on the reproducing kernel for the stated property to hold. In particular, we show that the DKSD detects non-convergence in moments with an appropriate kernel.
Introduction
In Bayesian statistics and machine learning, researchers frequently encounter "unnormalized" probability densities—distributions where the relative shape is known, but the normalizing constant (the total area under the curve) is computationally intractable. This intractability renders standard divergence measures, such as the Kullback-Leibler (KL) divergence, difficult to apply directly for model evaluation or MCMC diagnostics.
Kernel Stein Discrepancies (KSDs) emerged as a powerful solution to this problem. By leveraging Stein’s method, KSDs allow for the comparison of a sample set and a target distribution without requiring the normalization constant. However, a significant limitation has persisted: standard KSDs primarily control "weak convergence." While they can confirm if a sample set matches the general shape of a distribution, they often fail to detect errors in the "moments"—such as the mean and variance. For high-stakes applications in physics, finance, and engineering, ensuring the accuracy of these moments is often as critical as the overall distribution shape.
The Problem: The Moment Blind Spot
Previous research established that commonly used kernels, such as the Inverse Multi-Quadratic (IMQ) kernel, are mathematically incapable of guaranteeing that a sample's mean and variance are correct. This "blind spot" occurs because these kernels do not grow fast enough to capture discrepancies in the tails of a distribution. Consequently, a KSD might decay toward zero, suggesting convergence, even while the sample's second moment remains significantly biased.
Methodology: Kernel Tilting and Diffusion KSDs
This paper addresses these limitations by analyzing Diffusion KSDs (DKSD), a generalization of the standard Langevin KSD. The authors introduce two primary technical innovations to bridge the gap between KSDs and moment convergence:
- Kernel Tilting: The authors demonstrate that for a KSD to control the $q$-th moment, the kernel must grow at a specific rate relative to $q$. They introduce a "tilting" method where a base kernel is multiplied by weight functions that grow polynomially. This modification forces the KSD to become sensitive to the tails of the distribution, where moments are determined.
- Pseudo-Lipschitz Functions: To provide a rigorous theoretical foundation, the authors relate the DKSD to a class of "pseudo-Lipschitz" functions. These functions generalize the standard Lipschitz property by allowing for polynomial growth. By proving that the DKSD controls the integral probability metric (IPM) defined by these functions, the authors establish the first KSDs known to exactly characterize $q$-Wasserstein convergence.
Experimental Validation
As this is a theoretical statistics contribution, the authors utilized synthetic benchmarks to validate the mathematical properties of the proposed "tilted" kernels (e.g., the IMQ+ kernel):
- Multivariate Gaussian ($d=5$): Used to demonstrate how standard KSDs fail to detect bias in the second moment (variance), whereas the IMQ+ kernel correctly identifies non-convergence.
- Multivariate Student’s $t$-distribution: A heavy-tailed distribution used to show that standard KSDs can fail to detect errors even in the first moment (mean) when tails are thick. The Diffusion KSD with appropriate tilting successfully detected these errors.
- Gaussian Mixture Models: Used to explore the behavior of KSDs in the presence of isolated modes, revealing that while tilted kernels improve moment detection, they still face challenges when probability mass is distributed across distant "islands."
Key Insights
- Growth Rates are Crucial: If a kernel is bounded (like the Gaussian kernel), it will inherently be "blind" to moment non-convergence. Sensitivity to moments requires specific polynomial growth in the kernel.
- Wasserstein Equivalence: The work provides explicit upper bounds on the $q$-Wasserstein distance in terms of the new DKSD. This ensures that if the DKSD approaches zero, the samples are guaranteed to converge to the target in both shape and moments.
Limitations and Future Directions
Despite the theoretical advancement, several practical challenges remain:
- Dimensionality: The theoretical bounds exhibit a heavy dependence on the dimension $d$, suggesting that in very high-dimensional spaces, a large number of samples may be required for tight guarantees.
- Isolated Modes: KSDs can still be "blind" to missing components of a distribution if those components are far from the current samples. Adaptive kernel scaling or bandwidth tuning may be necessary to mitigate this.
- Computational Complexity: The KSD calculation scales quadratically ($O(n^2)$) with the number of samples. Future work will need to refine the integration of tilted kernels with approximation techniques like kernel thinning to make them more viable for large-scale applications.
Conclusion
By introducing tilted kernels and establishing a link to pseudo-Lipschitz functions, this research provides a rigorous framework for using Kernel Stein Discrepancies to control moments. This ensures that MCMC diagnostics and goodness-of-fit tests are not just measuring the "shape" of a distribution, but are also capturing the critical tail behaviors and moments necessary for reliable statistical inference.
Summary updated