Abstract
Kernel mean embeddings—integrals of a kernel with respect to a probability distribution—are essential in Bayesian quadrature, but also widely used in other computational tools for numerical integration or for statistical inference based on the maximum mean discrepancy. These methods often require, or are enhanced by, the availability of a closed-form expression for the kernel mean embedding. However, deriving such expressions can be challenging, limiting the applicability of kernel-based techniques when practitioners do not have access to a closed-form embedding. This paper addresses this limitation by providing a comprehensive dictionary of known kernel mean embeddings, along with practical tools for deriving new embeddings from known ones. We also provide a Python library that includes minimal implementations of the embeddings.
Introduction
In kernel-based machine learning and statistics, algorithms frequently require the integration of a kernel function against a probability distribution. This process yields two fundamental quantities: the kernel mean embedding ($K_P$) and its integral ($K_{PP}$). These mathematical constructs serve as the engines for diverse applications, including Bayesian quadrature (probabilistic numerical integration), generative model evaluation, and statistical two-sample testing via Maximum Mean Discrepancy (MMD).
Despite their utility, calculating these integrals analytically is notoriously tedious and often mathematically complex. Historically, known formulas have been scattered across decades of literature spanning machine learning, statistics, signal processing, and numerical analysis. This fragmentation often forces practitioners to either waste significant time re-deriving complex integrals or rely on computationally expensive and noisy sample-based approximations. This work bridges this gap by centralizing these formulas into a single, accessible reference.
Methodology
The paper serves as both a theoretical reference and a practical toolkit, structured around three main pillars:
- The Dictionary: A compilation of exact, closed-form mathematical expressions for $K_P$ and $K_{PP}$ across common probability distributions (e.g., Uniform and Gaussian) paired with popular kernels such as Gaussian, Matérn, Wendland, Brownian Motion, Power Series, and Spherical/Periodic Sobolev.
- Derivation Tools: A set of mathematical techniques to construct new closed-form embeddings from existing ones. These include product and sum rules for factorized or mixture distributions, change of variables, and change of measure tricks.
- Software Implementation: The release of
kernel_embedding_dictionary, an open-source Python library providing minimal, unit-tested implementations of these complex formulas.
Key Insights
Several key ideas emerge from this systematic consolidation:
- Centralization as Democratization: By bringing together formulas previously buried in obscure appendices, the authors significantly lower the barrier to entry for using exact kernel methods.
- Compositional Power: Complex embeddings do not always require bespoke derivations. By exploiting properties like factorized distributions (using product kernels) or mixture models (using sum kernels), sophisticated embeddings can be built from simple, known components.
- Stein Kernels as Alternatives: When closed-form embeddings are intractable, one can design kernels that bypass the integration. For instance, Langevin Stein kernels use the score function of a distribution to ensure their kernel mean embedding is exactly zero by definition.
Key Results
The primary contribution of this work is the comprehensive indexing of closed-form expressions. The authors successfully translated intricate mathematical equations—including the piecewise polynomial expansions required for Wendland kernels—into a functional Python package. This allows researchers to evaluate exact embeddings with minimal code.
Table 1 (above) serves as the central index for the dictionary, mapping kernel types and degrees to specific distributions and their corresponding closed-form references.
Limitations and Future Work
Analytical solutions have inherent limits; closed-form expressions do not exist for every kernel-distribution pair. For example, while the embedding for a Gaussian distribution with a Wendland kernel is provided, its integral ($K_{PP}$) remains uncomputable in closed form.
Future directions include expanding the dictionary to include integrals of products of kernels—useful for advanced Bayesian quadrature—and embeddings of conditional distributions. Where exact forms are unavailable, the community must continue to refine numerical approximations like Monte Carlo or Quasi-Monte Carlo estimators.
Summary updated