VIREL: A Variational Inference Framework for Reinforcement Learning.

Matthew Fellows, Anuj Mahajan, Tim G. J. Rudner, Shimon Whiteson

Abstract

Applying probabilistic models to reinforcement learning (RL) enables the uses of powerful optimisation tools such as variational inference in RL. However, existing inference frameworks and their algorithms pose significant challenges for learning optimal policies, e.g., the lack of mode capturing behaviour in pseudo-likelihood methods, difficulties learning deterministic policies in maximum entropy RL based approaches, and a lack of analysis when function approximators are used. We propose VIREL, a theoretically grounded probabilistic inference framework for RL that utilises a parametrised action-value function to summarise future dynamics of the underlying MDP, generalising existing approaches. VIREL also benefits from a mode-seeking form of KL divergence, the ability to learn deterministic optimal polices naturally from inference, and the ability to optimise value functions and policies in separate, iterative steps. In applying variational expectation-maximisation to VIREL, we thus show that the actor-critic algorithm can be reduced to expectation-maximisation, with policy improvement equivalent to an E-step and policy evaluation to an M-step. We then derive a family of actor-critic methods from VIREL, including a scheme for adaptive exploration. Finally, we demonstrate that actor-critic algorithms from this family outperform state-of-the-art methods based on soft value functions in several domains.

Introduction

Casting Reinforcement Learning (RL) as a probabilistic inference problem allows researchers to leverage robust variational inference tools for control tasks. However, current frameworks face notable limitations. Pseudo-likelihood methods often exhibit "risk-seeking" behavior by matching distribution averages rather than modes. Conversely, Maximum Entropy RL (MERL) methods, such as Soft Actor-Critic (SAC), struggle to achieve truly deterministic policies and are highly sensitive to temperature hyperparameters. VIREL (Variational Inference for Reinforcement Learning) addresses these issues by framing the actor-critic approach as a Variational Expectation-Maximization (EM) algorithm, enabling the natural learning of deterministic optimal policies and adaptive exploration.

The VIREL Framework

VIREL utilizes a parameterized action-value function ($Q$) to summarize the future dynamics of the environment. By treating RL as inference, the standard actor-critic loop is reformulated as a Variational EM procedure.

Comparison of graphical models for MERL and VIREL.
  • The E-Step (Actor): This step corresponds to policy improvement. The actor is updated to minimize the KL divergence between the current policy and a Boltzmann distribution derived from the $Q$-function. VIREL employs a mode-seeking form of KL divergence, focusing on the best actions rather than attempting to cover the entire action space.
  • The M-Step (Critic): This step corresponds to policy evaluation. The critic ($Q$-function) is updated to better represent expected rewards under the current policy.

A key innovation is the adaptive temperature ($\epsilon_\omega$), which is based on the Bellman residual error. As the critic's confidence increases and error decreases, the temperature drops, allowing the policy to transition from stochastic exploration to deterministic optimality.

Theoretical Insights

A major contribution of VIREL is the formal proof (Theorem 1) that as model error approaches zero, the Boltzmann distribution collapses into a Dirac delta function. This ensures that deterministic optimal policies are a natural mathematical consequence of the framework.

Counterexample MDP demonstrating sub-optimality in Maximum Entropy RL.

The authors also present a counterexample (illustrated above) showing that MERL frameworks can fail to find optimal policies. In certain MDPs, the over-prioritization of entropy in sub-optimal regions can prevent the agent from converging to the true optimum. In VIREL, the temperature represents the model's uncertainty about its $Q$-function, providing a principled mechanism for handling exploration without the pitfalls of fixed entropy regularization.

Experimental Evaluation

VIREL was evaluated against state-of-the-art baselines, including DDPG and SAC, on high-dimensional continuous control benchmarks from OpenAI Gym (MuJoCo) and rllab.

  • Performance in High Dimensions: While performance is comparable on simpler tasks like HalfCheetah, VIREL significantly outperforms baselines on complex tasks such as Humanoid-v2 and Ant-v2.
  • Stability: In the 376-dimensional state space of Humanoid-v2, DDPG failed to learn effectively, whereas VIREL variants achieved substantially higher average returns than SAC, demonstrating superior stability in high-dimensional spaces.

Conclusion

VIREL provides a unified theoretical framework that reconciles various RL methods (e.g., A3C, MPO) under a single inference-based umbrella. While the adaptive temperature requires careful estimation of Bellman residuals, it offers a more grounded approach to exploration. Future work may focus on improving sample efficiency through more advanced uncertainty-based exploration and extending the framework to multi-agent settings to address sub-optimality in collaborative tasks.

Summary updated

Abstract

Applying probabilistic models to reinforcement learning (RL) enables the uses of powerful optimisation tools such as variational inference in RL. However, existing inference frameworks and their algorithms pose significant challenges for learning optimal policies, e.g., the lack of mode capturing behaviour in pseudo-likelihood methods, difficulties learning deterministic policies in maximum entropy RL based approaches, and a lack of analysis when function approximators are used. We propose VIREL, a theoretically grounded probabilistic inference framework for RL that utilises a parametrised action-value function to summarise future dynamics of the underlying MDP, generalising existing approaches. VIREL also benefits from a mode-seeking form of KL divergence, the ability to learn deterministic optimal polices naturally from inference, and the ability to optimise value functions and policies in separate, iterative steps. In applying variational expectation-maximisation to VIREL, we thus show that the actor-critic algorithm can be reduced to expectation-maximisation, with policy improvement equivalent to an E-step and policy evaluation to an M-step. We then derive a family of actor-critic methods fromVIREL, including a scheme for adaptive exploration. Finally, we demonstrate that actor-critic algorithms from this family outperform state-of-the-art methods based on soft value functions in several domains.
BibTeX
@inproceedings{fellows2019virel,
  author = {Matthew Fellows and Anuj Mahajan and Tim G. J. Rudner and Shimon Whiteson},
  title = {{VIREL: A Variational Inference Framework for Reinforcement Learning.}},
  booktitle = {NeurIPS},
  year = {2019},
  url = {https://openreview.net/forum?id=1tWA8fYM9YO},
}
Figure