Abstract
In this paper, we study the statistical and geometrical properties of the Kullback-Leibler divergence with kernel covariance operators (KKL) introduced by Bach [2022]. Unlike the classical Kullback-Leibler (KL) divergence that involves density ratios, the KKL compares probability distributions through covariance operators (embeddings) in a reproducible kernel Hilbert space (RKHS), and computes the Kullback-Leibler quantum divergence. This novel divergence hence shares parallel but different aspects with both the standard Kullback-Leibler between probability distributions and kernel embeddings metrics such as the maximum mean discrepancy. A limitation faced with the original KKL divergence is its inability to be defined for distributions with disjoint supports. To solve this problem, we propose in this paper a regularised variant that guarantees that the divergence is well defined for all distributions. We derive bounds that quantify the deviation of the regularised KKL to the original one, as well as finite-sample bounds. In addition, we provide a closed-form expression for the regularised KKL, specifically applicable when the distributions consist of finite sets of points, which makes it implementable. Furthermore, we derive a Wasserstein gradient descent scheme of the KKL divergence in the case of discrete distributions, and study empirically its properties to transport a set of points to a target distribution.
Introduction
Measuring the distance between a model distribution $p$ and a target distribution $q$ is a fundamental task in machine learning. While the standard Kullback-Leibler (KL) divergence is theoretically robust, it often fails (diverges to infinity) when the supports of $p$ and $q$ are disjoint—a common scenario in discrete sampling or low-dimensional manifold learning. Conversely, the Maximum Mean Discrepancy (MMD) is computationally efficient but can lack geometric sensitivity, often getting trapped in local minima during optimization because it primarily considers first-order moments of embeddings.
The Kernel KL (KKL) divergence, introduced by Bach (2022), offers a middle ground by utilizing second-order information via covariance operators in a Reproducing Kernel Hilbert Space (RKHS). However, the original KKL shares the standard KL's sensitivity to disjoint supports. This work addresses this limitation by introducing a regularized variant.
Methods
The authors propose the Regularized KKL ($KKL_\alpha$), which compares the source distribution $p$ to a mixture $(1-\alpha)q + \alpha p$. This "skewed" approach ensures the divergence remains finite and well-defined for any $\alpha \in (0, 1)$.
Key technical contributions include:
- Theoretical Analysis: Proofs of monotonicity with respect to $\alpha$ and derivation of finite-sample bounds showing how quickly the divergence calculated from $n$ points converges to the true underlying value.
- Closed-Form Expression: For discrete particle clouds, the authors derive an explicit formula using Kernel Gram matrices. This is a major win for implementation, as it avoids the need for complex density estimations.
- Wasserstein Gradient Flow: The derivation of the first variation of the functional allows for particle transport along steepest descent paths using efficient off-the-shelf optimizers like L-BFGS.
Results and Insights
The $KKL_\alpha$ divergence demonstrates significant advantages over existing metrics. Unlike MMD, which relies on mean embeddings (first-order), KKL's use of covariance operators (second-order) makes it more discriminative, especially when distributions have the same mean but different shapes. Compared to the Kernelized Aligned Likelihood Estimator (KALE), $KKL_\alpha$ is much faster because it has a direct, analytical formula, avoiding the need to solve an internal optimization problem at every step.
Empirical evaluations on synthetic benchmarks, such as anisotropic Gaussians and non-overlapping ring structures, validate the statistical properties of the metric. As shown in the figure above, the empirical estimate of $KKL_\alpha$ converges to the population limit as the number of particles increases, following a rate of approximately $O(1/\sqrt{n})$. The metric also exhibits robustness to the choice of the regularization parameter $\alpha$, simplifying hyperparameter tuning. In distribution transport tasks, the gradient flow of $KKL_\alpha$ proves more "sticky" to target geometries than MMD, accurately capturing complex supports like the "3-rings" target.
Conclusion and Future Work
While $KKL_\alpha$ provides a robust and geometrically sensitive tool for distribution approximation, its current $O((n+m)^3)$ complexity due to singular value decompositions (SVD) limits its application to a few thousand particles. Future research will explore scaling techniques such as random features or Nyström approximations. Additionally, while empirical results for Wasserstein gradient flows are promising, a formal theoretical proof of their convergence remains an open challenge. Further study is also needed to evaluate the performance of $KKL_\alpha$ in very high-dimensional latent spaces.
Summary updated