A Dissipativity Theory for Undiscounted Markov Decision Processes

Sébastien Gros, Mario Zanon

Abstract

Economic Model Predictive Control (EMPC) relies on dissipativity theory to guarantee system stability under general economic objectives. However, this framework has historically been incompatible with stochastic systems modeled as Markov Decision Processes (MDPs). This paper identifies the fundamental mathematical barriers to applying standard dissipativity to MDPs and proposes a novel solution: lifting the dissipativity framework from the state space to the space of probability measures. By utilizing nonlinear storage functionals and statistical dissimilarity metrics, such as Kullback-Leibler (KL) divergence, the authors establish a new criterion for functional strict dissipativity that ensures the asymptotic stability of optimal policies in stochastic environments.

Introduction

In modern control engineering, Model Predictive Control (MPC) is a standard for steering systems toward desired setpoints. While traditional tracking MPC focuses on minimizing quadratic error, Economic MPC (EMPC) optimizes broader performance metrics like fuel efficiency, financial cost, or energy consumption. While EMPC offers superior performance, proving that the resulting system will remain stable is significantly more complex.

For deterministic systems, stability is typically proven using dissipativity theory. This involves finding a "storage function" to perform a "cost rotation"—a mathematical transformation that ensures the economic cost behaves like a standard Lyapunov function. However, extending this to stochastic systems, specifically Markov Decision Processes (MDPs), has remained an open challenge. This paper provides both a proof of why traditional methods fail and a new theoretical framework to overcome these limitations.

The Fundamental Incompatibility

The authors first establish a rigorous "impossibility result" regarding standard dissipativity. In deterministic systems, a linear cost rotation is often sufficient to prove stability. However, the authors prove (via Lemma 1) that for general continuous-state MDPs, standard linear cost rotations are mathematically incapable of yielding a valid Lyapunov function.

Because stochastic noise causes the system's state distribution to spread out over time, the standard rotated cost cannot be bounded strictly above zero. This breakdown in the mathematical framework means that traditional dissipativity cannot guarantee that a stochastic system will settle into an optimal steady state, as the noise prevents the cost function from satisfying the necessary stability criteria.

Methodology: Lifting to Measure Space

To resolve this incompatibility, the paper proposes a paradigm shift: instead of analyzing the stability of the system's state, researchers should analyze the stability of the probability measure (the distribution of states). This "lifting" of the problem into measure space involves three key innovations:

  1. Statistical Dissimilarity Measures: Rather than using Euclidean distance to measure the gap between the current state and the goal, the authors employ metrics like Kullback-Leibler (KL) divergence, the Wasserstein metric, or total variation distance. These measure the distance between the current probability distribution and the optimal steady-state distribution.
  2. Nonlinear Storage Functionals: The authors demonstrate that linear storage functions are insufficient for MDPs. They introduce nonlinear storage functionals, denoted as $\lambda[\rho]$, which operate on the entire probability measure rather than individual state points.
  3. Functional Strict Dissipativity: By combining these elements, the authors define a new "functional strict dissipativity" criterion. If an MDP satisfies this criterion, the closed-loop system is mathematically guaranteed to converge asymptotically to the optimal steady-state distribution.

Key Results and Validation

The primary theoretical contribution is Theorem 2, which provides the formal guarantee: if a bounded nonlinear storage functional exists, the optimal policy of the MDP is $D$-asymptotically stable. This ensures that the system's probability distribution will reliably converge to the optimal steady state despite the presence of noise.

To validate this theory, the authors applied it to a stochastic Linear Quadratic Regulator (LQR) subject to Gaussian noise. In this 2D simulated environment, they demonstrated that:

  • The traditional stage cost failed as a Lyapunov function, dipping into negative values.
  • The newly derived nonlinear rotated cost functional remained strictly positive.
  • The cost functional bounded the KL divergence and decreased monotonically over time, satisfying the requirements for Lyapunov stability.

Conclusion and Future Directions

This work establishes a foundational theoretical bridge between economic control theory and stochastic processes. While the current framework is limited to undiscounted, infinite-horizon MDPs with continuous state spaces, it opens several avenues for future research. The authors are currently extending this theory to discounted MDPs and finite-horizon stochastic MPC schemes. Furthermore, this framework has significant implications for Safe Reinforcement Learning, providing a mathematical basis for guaranteeing stability and safety in learning-based control systems.

Summary updated

BibTeX
@article{gros2022dissipativity,
  author = {Sébastien Gros and Mario Zanon},
  title = {{A Dissipativity Theory for Undiscounted Markov Decision Processes}},
  journal = {Automatica},
  year = {2022},
  doi = {10.1016/j.automatica.2022.110602},
}
Figure