In classical robotics and reinforcement learning, planning and estimation are treated as fundamentally distinct problems. Estimation looks backward, using past observations to infer the current hidden state of the world. Planning looks forward, searching for a sequence of actions that maximizes a scalar reward function.
However, a powerful mathematical duality exists between these two paradigms. Instead of treating planning as the maximization of an arbitrary reward, we can frame it as a problem of probabilistic inference. Under this view, an agent does not ask, "How can I maximize my reward?" Instead, it assumes a future state of success and asks, "Given that I successfully achieve my goal, what is the most likely sequence of actions that brought me here?"
By conditioning future trajectories on the event of success, sequential decision-making becomes equivalent to calculating a posterior distribution over actions. This paradigm shift unlocks the entire toolbox of statistical inference—such as Variational Inference (VI), Expectation-Maximization (EM), and Sequential Monte Carlo (SMC)—to solve complex control problems.
The Optimality Variable Trick
To transform control into inference, we must translate rewards into probabilities. As outlined by Sergey Levine et al. and Akira Kinose et al., this is achieved by introducing a binary random variable, $O_t$, which denotes whether the agent acts "optimally" at time $E_t$. We define the probability of optimality conditioned on a state $s_t$ and action $a_t$ as proportional to the exponential of the reward:
$$P(O_t = 1 \mid s_t, a_t) = \exp(r(s_t, a_t))$$
If we construct a dynamic graphical model where the state transitions follow the environment's passive dynamics, the problem of finding the optimal policy becomes equivalent to inferring the posterior distribution of the trajectory given that all optimality variables are true: $P(a_{1:T} \mid O_{1:T} = 1)$.
[ s_1 ] ----> [ s_2 ] ----> [ s_3 ] (State Dynamics)
| | |
v v v
[ O_1 ] [ O_2 ] [ O_3 ] (Conditioned on Optimality = 1)
As Deepak Kumar Verma et al. and Cameron E. Freer et al. demonstrate, this recursive probabilistic planning framework defines optimal behavior self-referentially. Current choices are forced to be mathematically consistent with future goal-oriented states. Daniel C McNamee argues that this exact duality explains how biological systems overcome sensory noise and transmission delays, working backward from desired outcomes to determine motor commands.
Why It Works: Bypassing the Intractability of Control
Framing control as inference is not just a philosophical reinterpretation; it resolves severe computational bottlenecks inherent to classical control.
1. Replacing PDEs with Parallelizable Sampling
In continuous-time stochastic optimal control, finding the exact optimal policy requires solving the non-linear Hamilton-Jacobi-Bellman (HJB) partial differential equation, which quickly becomes intractable due to the curse of dimensionality.
By mapping the control task to a variational inference problem, Yichen Wang et al. demonstrate that these complex differential equations can be bypassed entirely. Instead, the problem is transformed into finding an optimal probability measure over future trajectories, which can be solved using highly parallelizable Monte Carlo sampling.
2. Preserving Multimodality
Traditional trajectory optimization algorithms often collapse to a single point estimate or a single deterministic path. In complex, non-convex environments (such as a robot navigating around an obstacle), there may be multiple equally valid paths.
Masashi Okada proposes that reframing Model Predictive Control (MPC) as Bayesian variational inference unifies stochastic control methods under distribution matching. Rather than performing simple moment matching that collapses to an average, hazardous path, controllers can explicitly model the full posterior distribution.
To exploit this, Alexander Lambert et al. and Darrick Lee et al. utilize Stein Variational Gradient Descent (SVGD) to maintain and evolve a diverse distribution of trajectories in parallel, preventing the system from getting trapped in local minima.
/--- Path A (Left around obstacle) ---\
Start State --------< >---- Goal
\--- Path B (Right around obstacle) ---/
3. Decoupling Exploration from Policy Updates via EM
When reinforcement learning is framed as inference, the policy parameters can be optimized using the Expectation-Maximization (EM) algorithm. As shown by Matthew Hoffman et al., the E-step infers the reward-weighted posterior distribution over trajectories, while the M-step updates the policy parameters to match these high-performing paths.
This decoupling provides immense stability. For instance, Zuxin Liu et al. leverage this EM structure in Constrained Variational Policy Optimization (CVPO) to separate reward maximization from safety constraint satisfaction, yielding massive sample-efficiency gains. Similarly, Haozhe Wang et al. use EM-guided optimization to stabilize reinforcement learning in discrete decision spaces, alternating between structured exploration and policy refinement.
Unifying Perception and Action
When planning is inference, the boundary between perception (state estimation) and action (control) dissolves. Both can be optimized simultaneously within a single probabilistic graphical model.
Active Inference and Free Energy Minimization
In the active inference framework, target states are treated as strong prior beliefs within a generative model. The agent minimizes variational free energy, which simultaneously forces its internal states to match reality (perception) and forces its actions to fulfill its prior expectations of success (control).
Christopher Buckley et al. demonstrate that classical engineering tools like PID controllers are mathematically equivalent to active inference agents minimizing prediction errors under simple linear generative models. To scale this to complex systems, Cristian Meo et al. combine deep Multimodal Variational Autoencoders (VAEs) with active inference, allowing a high-dimensional robotic arm to adapt to sensory occlusions and unmodeled physical changes without parameter retuning.
Task-Oriented Representations
If state estimation and control are separate, a visual representation model might waste capacity encoding irrelevant background details. By optimizing both jointly as an inference problem, the system learns task-oriented representations.
Jung-Su Ha et al. establish that the duality between stochastic optimal control and variational inference allows a system to transition from state estimation to goal-directed control by simply swapping observation costs for task costs. This unified framework ensures that the learned latent representations prioritize features critical to the control objective, as corroborated by the sequential variational soft Q-learning networks developed by Shiyu Huang et al..
Algorithmic Implementations
How do we practically compute these reward-weighted posteriors? Several distinct algorithmic families have emerged:
- Sequential Monte Carlo (SMC): Alexandre Piché et al. frame planning within a hidden Markov model where the value function acts as a backward message. This allows the use of particle filters to sample continuous actions and track multimodal strategies. Hany Abdulsamad et al. extend this to continuous POMDPs, using non-Markovian models to sample from reward-weighted posteriors, drastically reducing gradient variance.
- Factor Graphs and Smoothing: Mustafa Mukadam et al. show that for continuous-time stochastic systems, the optimal control policy corresponds to the mode of the posterior trajectory distribution. This allows motion planning to be solved efficiently using sparse factor graph smoothing. Matthew King-Smith et al. build on this to integrate control, planning, and estimation into a single factor graph, causing the agent to naturally exhibit cautious, collision-avoiding behavior when sensor noise increases.
- Variational Trajectory Optimization: By treating trajectory optimization as Bayesian inference, algorithms can maintain a probability distribution over action sequences rather than a single point estimate. Guillaume Crevecoeur et al. utilize this to mathematically justify common heuristics like adaptive covariance and exploration noise. Joe Watson et al. leverage this framework to achieve precise covariance control and adaptive risk-seeking behavior.
Broader Dualities
The mathematical bridge between control and inference extends far beyond standard reinforcement learning:
- Goal-Conditioned RL without Rewards: Instead of manually shaping rewards, Tim G. J. Rudner et al. formulate goal-conditioned RL as minimizing the KL divergence between the agent's actual trajectory distribution and a target posterior distribution conditioned on an outcome. This automatically derives dense, data-driven reward signals from raw pixels, a concept mirrored in the event-driven inverse control framework by Justin Fu et al..
- Diffusion Models: Julius Berner establishes that the Evidence Lower Bound (ELBO) in variational inference is formally equivalent to the optimal cost in a stochastic control problem. This proves that sampling from a diffusion-based generative model is mathematically identical to finding the path of least energy in a controlled dynamical system.
- Experimental Design: Sahel Iqbal et al. demonstrate a duality between sequential Bayesian experimental design and risk-sensitive control. This allows researchers to apply specialized particle filtering and smoothing tools to optimize experimental policies, treating the search for informative experiments as a maximum likelihood estimation task.