In standard reinforcement learning, we typically think of an agent as a function that maps states to actions, optimized via trial and error to maximize a scalar reward. However, a powerful alternative perspective treats reinforcement learning not as a search for a controller, but as a problem of statistical inference.
By framing "optimality" as a latent variable we wish to infer, researchers have developed a family of algorithms—most notably those based on Expectation-Maximization (EM)—that transform the unstable process of policy gradients into a sequence of stable, supervised learning tasks.
The Core Intuition: Optimality as a Random Variable
The fundamental shift in this research direction is the introduction of a binary random variable, often denoted as $O$ (for "Optimality"). We define the probability of being optimal, $P(O=1)$, as being proportional to the reward received.
Under this lens, the goal of reinforcement learning is no longer just "maximizing reward," but rather computing the posterior distribution of actions given that we have observed a "success" event ($O=1$). As Marc Toussaint and Nikos Vlassis argue in Model-free reinforcement learning as mixture learning, this allows us to treat the value function as the likelihood of success, turning policy discovery into a maximum likelihood estimation problem.
The EM Framework for Policy Search
Once RL is framed as inference, the Expectation-Maximization (EM) algorithm becomes the natural engine for optimization. This approach generally follows a two-step iterative process:
- The E-Step (Expectation): The agent evaluates its current experience to identify which actions or trajectories were most consistent with "optimality." Instead of updating the policy parameters directly, it creates a non-parametric "target" distribution. This target distribution places higher weight on actions that yielded high rewards or advantages.
- The M-Step (Maximization): The agent updates its parametric policy (e.g., a neural network) to match this target distribution. Crucially, this step is a weighted supervised learning task.
This decomposition is the secret to the stability of these methods. Supervised learning is numerically robust and well-understood compared to the high-variance nature of policy gradients. Jan Peters and Stefan Schaal demonstrated this in Reinforcement learning by reward-weighted regression for operational space control, where they showed that framing control as EM allows high-degree-of-freedom robots to learn complex movements without needing a physics model.
Stability Through Trust Regions
A recurring challenge in RL is the "policy collapse" problem, where a single bad update destroys the agent's performance. The inference perspective solves this by naturally incorporating trust regions.
In Maximum a Posteriori Policy Optimisation (MPO), Abbas Abdolmaleki and colleagues at DeepMind use the EM framework to decouple the "what to learn" from the "how to learn." The E-step finds a soft-optimal action distribution, while the M-step uses a Kullback-Leibler (KL) divergence constraint to ensure the new policy doesn't stray too far from the old one. This makes MPO and its on-policy variant, V-MPO, significantly less finicky than standard methods like PPO, as they don't require the same level of hyperparameter tuning for entropy regularization.
Beyond Scalar Rewards: Preferences and Goals
The probabilistic framing is particularly flexible when dealing with complex feedback. Because the framework is built on the likelihood of "success," it can be adapted to learn from sources other than a single reward number:
- Sparse Goals: Alp Kucukelbir introduced Hindsight Expectation Maximization (hEM), which treats hindsight relabeling as importance sampling within an EM loop. This allows agents to learn from high-dimensional image inputs even when rewards are extremely sparse.
- Human Preferences: Recent work on PMPO (Preference-based MPO) by Abbas Abdolmaleki and Bilal Piot extends this to learning from negative feedback, or positive feedback or both. By treating preference as a probabilistic event, the model can learn from unpaired examples—even learning safely from failures alone.
Why It Works: Eliminating the Learning Rate
One of the most practical benefits of the EM-based approach is the reduction of manual tuning. In Policy search for motor primitives in robotics, Jens Kober and Jan Peters introduced PoWER, an algorithm that eliminates the need for a manually tuned learning rate. Because the M-step is an analytical update based on weighted averages of successful trajectories, the step size is effectively determined by the data itself. This allowed a real robot arm to learn the "Ball-in-a-Cup" task in fewer than 100 trials—a feat that would be nearly impossible with standard gradient-based RL.
A Unified View of Learning
Ultimately, this research direction points toward a "Standard Model" of machine learning. As Zhiting Hu suggests in Toward a 'Standard Model' of Machine Learning, reframing RL as inference allows us to treat rewards, data, and rules as a single "experience function." This unification suggests that the boundary between supervised learning and reinforcement learning is artificial; both are simply different ways of performing inference on the latent variables of optimal behavior.