In standard reinforcement learning, the Bellman optimality equation is fundamentally non-linear. This non-linearity arises from the $\max$ operator over discrete actions or the quadratic optimization of control inputs. While this structure is the bedrock of the field, it makes complex tasks difficult to decompose; you cannot simply "add" two optimal value functions together to solve a combined task.
A significant body of research explores a mathematical "shortcut" to this problem: by carefully choosing the cost structure and applying an exponential transformation to the value function, the non-linear Bellman equation collapses into a linear system.
The Cost of Deviation
The key to this linearization lies in how we define the cost of an action. In a standard Markov Decision Process (MDP), an agent chooses an action from a discrete set. In the framework of Linearly-Solvable MDPs (LMDPs), introduced by Emanuel Todorov in Linearly-solvable Markov decision problems, the agent instead modifies the transition probabilities of the environment directly.
The control cost is defined as the Kullback-Leibler (KL) divergence between the controlled transition distribution and a "passive" or baseline dynamics distribution. As noted by Todorov, this formulation makes the optimization problem convex. More importantly, when this KL-divergence cost is combined with an exponential mapping of the value function—transforming the optimal cost-to-go $V$ into a "desirability function" $z = \exp(-V)$—the Bellman equation becomes a linear eigenvalue problem.
Compositionality and Superposition
The most powerful intuition behind this transformation is compositionality. In linear systems, the principle of superposition applies: if you have a solution for Task A and a solution for Task B, their sum (or a weighted combination) is a solution for a combined task.
Saxe et al. leverage this in Hierarchy Through Composition with Multitask LMDPs to move away from traditional serial hierarchies (where a manager picks one subroutine at a time). Because the desirability functions are linear, multiple task-optimal controllers can be blended in parallel. As Todorov proves in Compositionality of optimal control laws, these state-dependent mixtures maintain global optimality, allowing complex behaviors to be synthesized from simple primitives without re-solving the entire control problem.
From Control to Path Integrals
In continuous state spaces, this transformation takes the form of the Cole-Hopf transformation, which converts the non-linear Hamilton-Jacobi-Bellman (HJB) equation into a linear partial differential equation.
This linearization enables the use of the Feynman-Kac formula, which expresses the solution to the PDE as an expectation over stochastic trajectories. Theodorou et al. utilize this in Learning Policy Improvements with Path Integrals to develop the PI² algorithm. Instead of computing gradients, the agent samples potential future paths and weights them by their exponentiated costs. This "path integral" approach allows for high-dimensional control (such as robotic limbs with many degrees of freedom) to be solved through Monte Carlo sampling rather than exhaustive grid-based methods.
Handling Risk and Robustness
The exponential transformation is not merely a computational trick for efficiency; it is deeply connected to how agents handle uncertainty. The "entropic risk measure"—which penalizes high-variance outcomes—naturally possesses an exponential structure.
Fei et al. show in Risk-Sensitive Reinforcement Learning with Function Approximation: A Debiasing Approach that by mapping value functions into a shifted exponential space, one can recover a linear structure that enables efficient regression even for non-linear risk objectives. Similarly, Zhang et al. demonstrate in Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity that this "Z-function" transformation allows practitioners to apply standard linear estimation techniques to solve complex offline reinforcement learning problems that would otherwise be computationally intractable.
By moving from the additive space of rewards to the multiplicative space of desirability, these methods bypass the "curse of non-linearity," enabling faster learning, easier task composition, and principled risk management.
Key Foundational Work
Linearly-solvable Markov decision problems (2006) — contributed 3 concept elements to this direction.
Compositionality of optimal control laws (2009) — contributed 2 concept elements to this direction.