Well-Posed KL-Regularized Control via Wasserstein and Kalman–Wasserstein KL Divergences

Viktor Stein, Adwait Datar, Nihat Ay

TL;DR

This paper introduces new "state-space-aware" KL divergences by replacing the standard Fisher-Rao geometry with transport-based geometries (Wasserstein, Kalman-Wasserstein). These new divergences are well-posed even in low-noise regimes where traditional KL regularization breaks down, leading to improved and more stable control policies in reinforcement learning.

What Problem Does This Paper Address?

Kullback-Leibler (KL) divergence is a popular tool for regularizing control policies in reinforcement learning and optimal control. It helps to keep the learned policy close to a reference policy or to ensure stability. However, the standard KL divergence has a critical flaw: it becomes infinite when the support of two probability distributions doesn't overlap (support mismatch) and degenerates in low-noise scenarios. This means that as the noise in a system approaches zero, the KL penalty can become infinitely large, leading to singular control problems and unstable or poorly performing policies. This paper aims to address this "low-noise pathology" by developing new divergence measures that are robust to vanishing noise and support mismatch, while still retaining the desirable properties of KL regularization.

Methods

The core idea is to re-interpret the KL divergence through an information-geometric lens. The KL divergence can be seen as a geodesic distance on a statistical manifold, weighted by the Fisher-Rao metric. This paper proposes replacing the Fisher-Rao geometry with other, "state-space-aware" geometries, specifically those derived from optimal transport theory like the Wasserstein-2 geometry and its linearized and Kalman variants. By using these new geometries, the authors derive new divergence functionals, termed "state-space-aware KL divergences." They then derive closed-form expressions for these divergences between common distribution families, particularly elliptic distributions (which include Gaussians). Finally, they demonstrate the utility of these new divergences by applying them to KL-regularized optimal control problems, specifically in the context of linear time-invariant systems with Gaussian process noise.

Datasets

Not discussed in the paper. The paper focuses on theoretical development and simulation-based examples rather than training on specific datasets.

Key Insights and Ideas

  • KL Divergence's Low-Noise Pathology: The paper clearly demonstrates and explains why standard KL divergence regularization fails in low-noise systems, leading to singular control problems. This is a crucial practical issue in many control applications.
  • Geometric Interpretation of Divergences: By framing divergences as geodesic distances on statistical manifolds, the paper provides a powerful geometric perspective. This allows for systematic construction of new divergences by changing the underlying geometry.
  • State-Space Awareness: The proposed Wasserstein and Kalman-Wasserstein divergences incorporate information about the geometry of the state space, unlike the classical KL divergence. This makes them more robust to issues like support mismatch and better suited for systems with varying noise levels.
  • Well-Posedness in Low Noise: The key takeaway is that these new divergences remain finite and well-behaved as system noise vanishes, leading to stable and effective control policies where KL-regularized methods fail.
  • Connection to Kalman Filters: The Kalman-Wasserstein divergence has a direct connection to techniques used in Kalman filtering, suggesting a natural bridge between these two important areas.

Key Results

  • New Divergences: The paper introduces and defines state-space-aware KL divergences based on Wasserstein, linearized Wasserstein, and Kalman-Wasserstein geometries.
  • Closed-Form Expressions: Explicit formulas for these new divergences are derived for elliptic distributions, including Gaussians.
  • Improved Optimal Control: In linear quadratic regulator (LQR) problems with Gaussian process noise, the proposed divergences (WKL and KWKL) lead to well-posed control problems, especially in low-noise regimes.
  • Better Performance: Simulations on a double integrator and a cart-pole system show that controllers regularized with WKL and KWKL exhibit better closed-loop performance, maintaining nontrivial feedback and avoiding oscillations seen with KL regularization as noise decreases.
  • Interpolation Behavior: The Kalman-Wasserstein KL divergence (KWKL) is shown to smoothly interpolate between the behavior of KL (high noise) and WKL (low noise) divergences, offering a flexible regularization approach.

Limitations and Future Work

  • Existence/Uniqueness of Geodesics: The paper acknowledges that criteria for ensuring the existence or uniqueness of geodesics for these new divergences are not provided. They conjecture that some pairs of elliptic distributions with different generators might not be connected by an O-dual geodesic, which could complicate the description of the density manifold's connected components.
  • Computational Approximation: For distributions outside of structured families, computing these G-divergences generally requires approximation. The authors suggest approaches like sampling-based estimation, local approximations, and plug-in methods.
  • Scalability: Developing scalable estimators and optimization algorithms for nonlinear Markov decision processes using these divergences is identified as an important direction for future work.
  • New Regularizers: The authors plan to explore new versions of mutual information based on these divergences and apply them to problems in imaging and Variational Autoencoders (VAEs).

Summary updated

Abstract

Kullback-Leibler (KL) divergence regularization is widely used in reinforcement learning, but it becomes infinite under support mismatch and can degenerate in low-noise regimes. Using a unified information-geometric framework, we introduce KL analogs by replacing the Fisher–Rao geometry in the dynamical formulation of the KL with transport-based geometries, and derive closed-form expressions for common distribution families. Between elliptic distributions, these divergences remain finite for degenerating equal covariances and yield a geometric interpretation of regularization heuristics used in Kalman ensemble methods. We demonstrate the utility of these divergences in KL-regularized optimal control. In the fully tractable setting of linear time-invariant systems with Gaussian process noise, the classical KL reduces to a quadratic control penalty that becomes singular as process noise vanishes. Our variants remove this singularity and yield well-posed problems. On a double integrator and a cart-pole example, the resulting controls preserve nontrivial feedback and achieve better closed-loop performance.
BibTeX
@inproceedings{stein2026well,
  author = {Viktor Stein and Adwait Datar and Nihat Ay},
  title = {{Well-Posed KL-Regularized Control via Wasserstein and Kalman–Wasserstein KL Divergences}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=diF53wYIj3},
}
Figure