VIREL: A Variational Inference Framework for Reinforcement Learning

Matthew Fellows, Anuj Mahajan, Tim G. J. Rudner, Shimon Whiteson

Abstract

Applying probabilistic models to reinforcement learning (RL) enables the application of powerful optimisation tools such as variational inference to RL. However, existing inference frameworks and their algorithms pose significant challenges for learning optimal policies, e.g., the absence of mode capturing behaviour in pseudo-likelihood methods and difficulties learning deterministic policies in maximum entropy RL based approaches. We propose VIREL, a novel, theoretically grounded probabilistic inference framework for RL that utilises a parametrised action-value function to summarise future dynamics of the underlying MDP. This gives VIREL a mode-seeking form of KL divergence, the ability to learn deterministic optimal polices naturally from inference and the ability to optimise value functions and policies in separate, iterative steps. In applying variational expectation-maximisation to VIREL we thus show that the actor-critic algorithm can be reduced to expectation-maximisation, with policy improvement equivalent to an E-step and policy evaluation to an M-step. We then derive a family of actor-critic methods from VIREL, including a scheme for adaptive exploration. Finally, we demonstrate that actor-critic algorithms from this family outperform state-of-the-art methods based on soft value functions in several domains.

Introduction

Framing Reinforcement Learning (RL) as a probabilistic inference problem has led to the development of highly successful algorithms like Soft Actor-Critic (SAC). These methods typically operate within the Maximum Entropy RL (MERL) framework, where agents are incentivized to maximize both cumulative rewards and policy entropy. While effective for exploration, MERL faces a fundamental limitation: the resulting optimal policies are inherently stochastic. In many real-world scenarios, the true optimal policy is deterministic. To approximate determinism, practitioners must manually tune a temperature hyperparameter—a process that is often brittle and fails to scale to high-dimensional environments with sparse or sub-optimal rewards.

The VIREL Framework

To address these limitations, we introduce VIREL (Variational Inference Framework for Reinforcement Learning). The core innovation of VIREL lies in its treatment of policy stochasticity. Instead of a fixed entropy bonus, VIREL utilizes a Boltzmann policy where the "temperature" is dynamically determined by the residual Bellman error of the action-value (Q) function.

Probabilistic graphical models for MERLIN and VIREL frameworks.

In this framework, when the Q-function is inaccurate (high error), the policy remains stochastic to encourage exploration. As the Q-function converges and the error approaches zero, the policy naturally collapses into a deterministic Dirac-delta distribution. This allows the agent to transition seamlessly from exploration to pure exploitation without manual hyperparameter scheduling.

Furthermore, VIREL provides a rigorous theoretical unification of Actor-Critic methods and Variational Expectation-Maximization (EM). We demonstrate that standard RL updates map directly to the EM process:

  • E-Step (Expectation): Corresponds to Policy Improvement (Actor training).
  • M-Step (Maximization): Corresponds to Policy Evaluation (Critic training).

Experimental Evaluation

We evaluated VIREL against state-of-the-art methods on standard continuous control benchmarks, including OpenAI Gym MuJoCo (Half-Cheetah, Ant, Humanoid) and rllab. These tasks range from low-dimensional locomotion to highly complex, high-dimensional control problems.

A structured MDP used to analyze exploration challenges in reinforcement learning.

Our results indicate that VIREL significantly outperforms SAC in high-dimensional tasks like Humanoid-v2 and Ant-v2. In these environments, the traditional entropy bonus in MERL can overwhelm sparse rewards, leading to sub-optimal behavior. VIREL’s use of "hard" value functions and error-based exploration avoids this pitfall. Conversely, in simpler tasks like Half-Cheetah, VIREL maintains parity with existing state-of-the-art methods, demonstrating its robustness across varying levels of complexity.

Conclusion and Future Work

VIREL offers a principled approach to the exploration-exploitation tradeoff by tying policy randomness to model uncertainty. While calculating the exact gradient of the residual error remains computationally challenging, empirical results using direct approximations are highly promising. Future research will focus on developing more tractable error estimates and extending the VIREL framework to Multi-Agent Reinforcement Learning (MARL) to address sub-optimality in collaborative and competitive settings.

Summary updated

Abstract

Applying probabilistic models to reinforcement learning (RL) enables the application of powerful optimisation tools such as variational inference to RL. However, existing inference frameworks and their algorithms pose significant challenges for learning optimal policies, e.g., the absence of mode capturing behaviour in pseudo-likelihood methods and difficulties learning deterministic policies in maximum entropy RL based approaches. We propose VIREL, a novel, theoretically grounded probabilistic inference framework for RL that utilises a parametrised action-value function to summarise future dynamics of the underlying MDP. This gives VIREL a mode-seeking form of KL divergence, the ability to learn deterministic optimal polices naturally from inference and the ability to optimise value functions and policies in separate, iterative steps. In applying variational expectation-maximisation to VIREL we thus show that the actor-critic algorithm can be reduced to expectation-maximisation, with policy improvement equivalent to an E-step and policy evaluation to an M-step. We then derive a family of actor-critic methods from VIREL, including a scheme for adaptive exploration. Finally, we demonstrate that actor-critic algorithms from this family outperform state-of-the-art methods based on soft value functions in several domains.
BibTeX
@inproceedings{fellows2019virel,
  author = {Matthew Fellows and Anuj Mahajan and Tim G. J. Rudner and Shimon Whiteson},
  title = {{VIREL: A Variational Inference Framework for Reinforcement Learning}},
  booktitle = {NeurIPS},
  year = {2019},
  url = {https://openreview.net/forum?id=1tWA8fYM9YO},
}
Figure