Stein operators, kernels and discrepancies for multivariate continuous distributions

Guillaume Mijoule, Gesine Reinert, Yvik SwanarXiv 2018 · DOI · arXiv

Abstract

In this paper we present a general framework for Stein's method for multivariate continuous distributions. The approach gives a collection of Stein characterisations, among which we highlight score-Stein operators and kernel Stein operators. Applications include copulas and distance between posterior distributions. We give a general construction for Stein kernels for elliptical distributions and discuss Stein kernels in generality, highlighting connections with Fisher information and mass transport. Finally, a goodness-of-fit test based on Stein discrepancies is given.

Introduction

Stein’s method is a powerful technique in probability theory used to bound the distance between two probability distributions. While the method is well-established for univariate (one-dimensional) cases, a general and tractable approach for multivariate continuous distributions has remained elusive. Most existing research focuses on specific cases, such as the multivariate Gaussian or Dirichlet distributions. However, researchers in fields like Bayesian statistics often deal with "intractable" multivariate densities where the normalization constant—the denominator in Bayes' rule—is unknown.

This paper addresses the need for a systematic, general framework to construct Stein operators and kernels for a wide variety of multivariate densities. By introducing "canonical" Stein operators and explicit constructions for Stein kernels, the authors create a toolkit for bounding distances between complex distributions and performing goodness-of-fit tests without requiring knowledge of normalization constants.

Methodology

The framework is built upon a fundamental building block termed the canonical Stein derivative. This directional derivative naturally produces functions with a mean of zero under a target distribution. From this foundation, the authors derive several "standardized" versions:

  1. Score-Stein Operators: These are based on the gradient of the log-density (the "score"). Because they rely on derivatives of the log-density, they are invariant to scaling, making them perfectly suited for Bayesian posterior analysis where the normalization constant is often the primary computational hurdle.
  2. Stein Kernels: A central contribution is the generalization of matrix-valued Stein kernels to higher dimensions. These functions satisfy a specific integration-by-parts identity and provide a path to quantifying distributional distances.
  3. Elliptical Distributions: The authors provide a general, explicit construction for Stein kernels for the entire family of elliptical distributions, which includes Gaussian, Student-t, and power exponential distributions. They demonstrate that for any elliptically symmetric distribution, the complex multivariate Stein kernel can be reduced to a relatively simple one-dimensional integral.
  4. Stein Discrepancies: These operators define discrepancies (such as the Kernelized Stein Discrepancy) that allow for comparing a sample to a target distribution directly.

Key Insights and Theoretical Connections

The paper highlights a deep link between Stein kernels and optimal transport. The authors show that the Stein kernel is essentially related to the map that "transports" one distribution to another, providing a geometric interpretation of an otherwise abstract probabilistic tool. Additionally, they derive bounds for the distance between "nested" distributions (where one is a reweighted version of the other), which has immediate applications in sensitivity analysis for Bayesian priors.

Results and Validation

As a theoretical work, the paper validates its framework using simulated data across several contexts:

  • Closed-form Kernels: The paper provides the first explicit, general formulas for Stein kernels for the multivariate Student-t and power exponential distributions.
  • Wasserstein Bounds: The authors provide concrete, quantitative bounds for the (1-)Wasserstein distance between various multivariate distributions, such as the distance between a skew-normal and a standard normal distribution. They also use copulas to demonstrate bounds on the Wasserstein distance between dependent and independent random variables.
  • Goodness-of-Fit Testing: A "Stein-based" goodness-of-fit test was implemented for the Student-t distribution. While currently a proof of concept, it offers a path forward for testing high-dimensional distributions where classical tests, like the Kolmogorov-Smirnov test, often fail.

Limitations and Future Work

The authors acknowledge that their initial implementation of the multivariate goodness-of-fit test currently possesses lower statistical power than some classical methods. They suggest that optimizing the choice of "kernels" (in the machine learning sense) within their framework could improve performance. Furthermore, while the framework is general, proving the existence and regularity of solutions to the "Stein equation" remains a challenge for distributions that are not log-concave. Finally, applying these methods to extremely high-dimensional data (e.g., thousands of dimensions) will require further optimization of the kernelized discrepancy calculations.

Summary updated

Abstract

In this paper we present a general framework for Stein's method for multivariate continuous distributions. The approach gives a collection of Stein characterisations, among which we highlight score-Stein operators and kernel Stein operators. Applications include copulas and distance between posterior distributions. We give a general construction for Stein kernels for elliptical distributions and discuss Stein kernels in generality, highlighting connections with Fisher information and mass transport. Finally, a goodness-of-fit test based on Stein discrepancies is given.
BibTeX
@misc{mijoule2018stein,
  author = {Guillaume Mijoule and Gesine Reinert and Yvik Swan},
  title = {{Stein operators, kernels and discrepancies for multivariate continuous distributions}},
  year = {2018},
  eprint = {1806.03478},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.03478},
}
Figure