At first glance, Reinforcement Learning (RL) and Variational Inference (VI) appear to solve entirely different problems. RL is about an agent taking actions to maximize an external cumulative reward within an environment. VI is about approximating an intractable probability distribution by maximizing a mathematical lower bound. Yet, a profound mathematical equivalence links these two domains: maximizing rewards under entropy regularization is formally identical to minimizing the divergence between an approximate policy and an optimal posterior distribution.
This duality has sparked a rich research direction that unifies inference and control. By treating sequential decision-making as probabilistic inference—and vice versa—we can bypass traditional computational bottlenecks, stabilize representation learning, and scale complex estimation tasks.
The Core Intuition: Mapping Rewards to Probabilities
To understand why this works, consider the objective functions of both fields. In maximum-entropy RL, the agent maximizes expected rewards alongside the policy's entropy to encourage exploration:
$$\mathcal{J}(\pi) = \mathbb{E}_{\pi} \left[ \sum_t R(s_t, a_t) + \alpha \mathcal{H}(\pi(\cdot|s_t)) \right]$$
In variational inference, we approximate a true posterior distribution $p(z|x)$ using a tractable distribution $q(z)$ by maximizing the Evidence Lower Bound (ELBO):
$$\text{ELBO}(q) = \mathbb{E}_{q}[\log p(x, z)] + \mathcal{H}(q)$$
If we map the latent variables $z$ to trajectories or actions, the approximate posterior $q$ to the policy $\pi$, and the joint log-likelihood $\log p(x, z)$ to the reward function $R$, the two objectives align perfectly. This structural symmetry allows us to move fluidly between inference and control in three distinct ways.
Variational Inference Reinforcement Learning
┌──────────────────────────────┐ ┌──────────────────────────────┐
│ Latent Variable (z) │ ◄────► │ Trajectory / Action (a) │
│ Approximate Posterior q(z) │ ◄────► │ Policy π(a|s) │
│ Log Joint Density log p(x,z)│ ◄────► │ Reward Function R(s,a) │
│ Maximize ELBO │ ◄────► │ Maximize Entropy-Reg. Return│
└──────────────────────────────┘ └──────────────────────────────┘
1. Control as Inference: Structuring Policies and Environments
When we view RL through the lens of inference, the policy becomes an approximate posterior over "successful" trajectories. This perspective provides a principled framework for learning latent representations and handling complex action spaces.
For instance, in model-based RL, agents often suffer from an "objective mismatch" where the predictive model is trained for maximum likelihood accuracy but utilized for downstream control. To fix this, Benjamin Eysenbach et al. introduced the MnM framework in Mismatched No More: Joint Model-Policy Optimization for Model-Based RL. By framing the entire trajectory as a hidden variable and the reward as a success probability, the authors derived a single, unified ELBO that jointly optimizes both the transition model and the policy under a single objective.
Similarly, Alex X. Lee et al. developed the Stochastic Latent Actor-Critic (SLAC) algorithm, which integrates latent variable modeling with policy optimization into a mathematically consistent maximum-entropy framework. This ensures that the learned representations are explicitly tailored for decision-making.
In partially observable or non-stationary settings, this joint optimization helps anchor latent spaces to what matters for long-term rewards, as demonstrated by Maximilian Igl et al. in Deep Variational Reinforcement Learning for POMDPs and Annie Xie et al. in Deep Reinforcement Learning amidst Continual Structured Non-Stationarity. By decoupling representation learning via variational reconstruction from the immediate policy updates, these methods stabilize training and allow agents to track environmental shifts independently of the immediate reward signal.
Furthermore, treating the policy as a generative model allows agents to capture complex, multimodal behaviors. Zhiao Huang leveraged this in Reparameterized Policy Learning for Multimodal Trajectory Optimization, formulating a variational lower bound that forces different latent variables to map to distinct, identifiable trajectory behaviors. To bridge the "amortization gap"—where neural network policies fail to capture the true optimal action distribution—Duo Xu et al. incorporated Hamiltonian Monte Carlo (HMC) steps directly into the ELBO optimization process in Improving Actor-Critic Reinforcement Learning via Hamiltonian Monte Carlo Method, evolving actions based on the Q-function's landscape to achieve a more accurate posterior.
2. Amortized Variational Reward Inference
Inverse Reinforcement Learning (IRL) aims to recover an underlying reward function from expert demonstrations. Traditional IRL requires an expensive "inner loop"—repeatedly solving a full RL problem to evaluate how well a candidate reward function explains the expert behavior. Variational inference offers an elegant escape hatch: treat the reward function itself as a latent variable and approximate its posterior.
In Scalable Bayesian Inverse Reinforcement Learning, Alex James Chan et al. introduced AVRIL, a framework that optimizes a variational lower bound to learn both a reward distribution and a policy simultaneously. Because it bypasses the inner-loop simulator entirely, it scales efficiently to high-dimensional offline datasets where simulators are unavailable.
Traditional IRL:
[Propose Reward] ──► [Run Full RL Loop (Simulator)] ──► [Compare to Expert] ──┐
▲ │
└─────────────────────────── Update Reward ────────────────────────────┘
Amortized Variational IRL (AVRIL / MAVRL):
[Expert Data] ──► [Shared Encoder (Neural Network)] ──► [Simultaneous Reward & Policy]
This concept is extended to handle diverse, multi-modal human feedback by Raphaël Baur et al. in MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inference. MAVRL uses an amortized variational encoder to map state transitions directly to reward distributions, satisfying constraints from demonstrations, preferences, and safety stops simultaneously without manual loss balancing. To capture the full uncertainty and risk profiles of expert behavior, Feiyang Wu et al. proposed Distributional Inverse Reinforcement Learning, using VI to learn the parameters of an entire reward distribution rather than just its mean.
3. Inference as Control: Solving Probabilistic Models via Policy Search
The duality also runs in reverse: complex, gradient-free probabilistic inference problems can be reframed and solved as reinforcement learning tasks. Here, the log-density of a target distribution acts as the reward signal, and a policy search algorithm acts as the optimizer.
In Efficient Gradient-Free Variational Inference using Policy Search, Oleg Arenz et al. introduced VIPS, an algorithm that treats the log-density of a target distribution as a reward function. By using information-geometric trust regions from policy search, VIPS updates Gaussian Mixture Models to explore and exploit high-probability regions without requiring target gradients.
Similarly, Arjun Karuvally et al. applied this in Learning proposals for sequential importance samplers using reinforced variational inference, treating the search for an importance sampling proposal distribution as a policy optimization problem where the log-likelihood ratio serves as the reward. For online state estimation, Andrew Campbell et al. demonstrated in Online Variational Filtering and Parameter Learning that framing online VI as an RL problem allows agents to use Bellman-type value function approximations to summarize historical observations into a compact, fixed-size representation, achieving constant-time updates per observation.
Summary
Ultimately, the integration of variational inference and reinforcement learning works because both fields are fundamentally concerned with optimization under uncertainty. Whether an algorithm is maximizing an ELBO to find a latent state or optimizing a policy to maximize expected returns, it is navigating a trade-off between exploration (entropy) and exploitation (likelihood or reward). By unifying these frameworks, the amortized efficiency of deep generative models can be directly injected into the sequential decision-making loops of RL, creating more robust, sample-efficient, and scalable intelligent systems.
Key Foundational Work
Learning proposals for sequential importance samplers using reinforced variational inference (2019) — contributed 2 concept elements to this direction.
Randomized Value Functions via Multiplicative Normalizing Flows (2019) — contributed 1 concept elements to this direction.
Efficient Gradient-Free Variational Inference using Policy Search (2018) — contributed 1 concept elements to this direction.