Optimality Conditioned Trajectory Inference

This direction is merged into another direction: control as inference. Its last version stays readable and citable here.

Version 1 of 131 papers41 conceptsfrozen 2026-10-04

In the standard framing of reinforcement learning (RL), an agent’s goal is to find a policy that maximizes a scalar sum of rewards. While intuitive, this "reward maximization" perspective often treats the agent as a black-box optimizer. A powerful alternative has emerged that reframes the problem entirely: Control as Inference.

In this framework, we do not ask "how do I maximize reward?" but rather "what must the agent's behavior look like, given that it is successful?" By treating optimal behavior as a latent variable to be inferred, researchers have unified reinforcement learning with the rich mathematical toolkit of Bayesian statistics and generative modeling.

The Optimality Variable

The core insight, popularized by Sergey Levine and Vladlen Koltun in Variational Policy Search via Trajectory Optimization, is the introduction of a binary random variable $O$ representing "optimality."

We define the probability of being optimal at any time step $t$ as proportional to the reward: $p(O_t=1 | s_t, a_t) = \exp(r(s_t, a_t))$. Under this view, the control problem is transformed into a probabilistic inference task: we want to compute the posterior distribution over trajectories $\tau$, conditioned on the event that we are optimal at every step: $p(\tau | O_{1:T}=1)$.

This shift changes the objective from finding a single optimal point to finding a distribution of successful behaviors. As Kevin P. Murphy notes in Reinforcement Learning: An Overview, this bridges the gap between generative modeling and policy optimization, treating the agent as a probabilistic model that "generates" high-reward actions.

Policy Search as Variational Inference

Because the true posterior $p(\tau | O=1)$ is usually intractable, we use Variational Inference (VI) to approximate it. We introduce a parameterized policy $q(\tau)$ and attempt to minimize the Kullback-Leibler (KL) divergence between our policy and the target posterior.

Andreas Krause and Jonas Hübotter demonstrate in Probabilistic Artificial Intelligence that this minimization is mathematically equivalent to maximizing the Evidence Lower Bound (ELBO). Crucially, this objective decomposes into two terms:

  1. Expected Reward: The standard RL objective.
  2. Entropy: A term that encourages the policy to be as diverse as possible.

This explains why "entropy-regularized" RL works so well. It isn't just a heuristic for exploration; it is the natural result of treating control as a Bayesian inference problem. As Beren Millidge and colleagues argue in Reinforcement Learning as Iterative and Amortised Inference, standard algorithms like policy gradients are actually just specific ways of optimizing this variational bound.

Solving Control with Expectation-Maximization

Reframing RL as inference allows us to use the Expectation-Maximization (EM) algorithm, a staple of unsupervised learning. In the "E-step," we estimate the distribution of trajectories that lead to reward; in the "M-step," we update the policy to match those trajectories.

This approach is particularly stable. Abbas Abdolmaleki and the DeepMind team used this in V-MPO to solve complex humanoid locomotion tasks. By using EM-style updates instead of standard policy gradients, they avoided the "policy collapse" common in high-dimensional continuous control.

Similarly, Joe Watson and Jan Peters developed Input Inference for Control (I2C), which treats control inputs as latent variables. This allows the agent to use message-passing techniques to "smooth" its plan over time, leading to much better performance in noisy, stochastic environments.

Why the Inference Perspective Matters

The "Control as Inference" framework provides three distinct advantages over traditional RL:

  1. Interpretability: By treating the policy as a posterior, we can decompose decisions into contributing factors. Qiuchen Qian and David Boyle show in their work on AWaVO that this probabilistic framing allows researchers to "peek under the hood" and see how specific constraints (like safety or wind) influence an agent's choices.
  2. Inverse Reinforcement Learning: If control is inference, then learning a reward function is simply Inverse Inference. Justin Fu and Sergey Levine demonstrated in VICE that an agent can learn to perform tasks by simply being shown what a "successful" state looks like, treating the goal as an event to be inferred.
  3. Unified Planning and Learning: In Variational Inference MPC, Masashi Okada shows that Model Predictive Control (MPC) and policy learning are just two sides of the same coin: one is iterative inference (planning), and the other is amortized inference (learning).

By moving from "maximizing rewards" to "inferring success," the field has gained a rigorous mathematical foundation that unifies how agents think, learn, and act.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_optimality_conditioned_trajectory_infere_7021efb9,
  title = {Optimality Conditioned Trajectory Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimality-conditioned-trajectory-inference/rdr_7021efb909534c308575bf2d8daf1960},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_optimality_conditioned_trajectory_infere_7021efb9_v1,
  title = {Optimality Conditioned Trajectory Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimality-conditioned-trajectory-inference/rdr_7021efb909534c308575bf2d8daf1960/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}