Soft Q-Learning with Mutual-Information Regularization

Jordi Grau-Moya, Felix Leibfried, Peter VrancxICLR 2019 · OpenReview

TL;DR

This paper introduces MIRL (Mutual-Information Reinforcement Learning), an algorithm that moves beyond standard entropy regularization by dynamically learning which actions are actually useful. By optimizing a "prior" distribution over actions, the agent focuses its exploration on high-impact behaviors, leading to a 30% performance boost over standard Soft Q-Learning on Atari benchmarks.

The Problem: The Flaw of Uniform Exploration

In Reinforcement Learning (RL), agents must balance "exploitation" (doing what works) with "exploration" (trying new things). A popular modern approach is Maximum Entropy RL, which rewards the agent for having a diverse policy. This prevents the agent from collapsing into a single predictable behavior too early.

However, there is a catch: standard entropy regularization assumes a uniform prior. It treats every possible action as equally worthy of exploration. In complex environments, many actions are useless or even detrimental most of the time. If an agent spends equal effort exploring "move toward the goal" and "stare at a wall," it wastes precious training time. Exploration should ideally be "informed" by what the agent has already learned about the general importance of specific actions across the entire environment.

Methods: Mutual-Information Reinforcement Learning

The authors propose a framework that treats RL as an inference problem where the prior action distribution is not fixed, but subject to optimization.

  1. Mutual-Information Regularization: Instead of just maximizing entropy (which pushes the policy toward a uniform distribution), MIRL maximizes an objective that includes a penalty on the Mutual Information (MI) between states and actions.
  2. The MIRL Algorithm: The core of the approach is a two-part update cycle:
  3. Policy Update: The agent learns a "Soft" Q-function that accounts for the current action prior.
  4. Prior Update: The agent updates its "prior" to match the marginal distribution of its actions. Essentially, if the agent finds itself jumping often to succeed, the prior for "Jump" increases.
  5. Adaptive Temperature: The method uses an adaptive inverse temperature parameter that scales based on the Q-function's loss. This allows the agent to explore heavily early on and become more deterministic as it gains confidence.

This approach effectively creates an Information Bottleneck. The agent tries to achieve high rewards while using as few "bits" of information as possible to deviate from its average (marginal) behavior.

Environments and Key Insights

The researchers evaluated MIRL on two main types of environments: simple 2D Grid Worlds (to provide intuition on how the prior affects exploration) and the Atari 2600 suite (as a high-dimensional benchmark).

Grid World Experiments
  • From Entropy to MI: The core insight is that optimizing the prior in an entropy-regularized framework mathematically leads to mutual-information regularization. This connects RL to Rate-Distortion Theory from information science.
  • Exploration with a "Hunch": By learning a prior, the agent develops a "hunch" about which actions are generally good. In a game like RoadRunner, if running right is usually better than running left, the agent will prioritize exploring variations of "running right" rather than wasting time running into a wall.
RoadRunner Prior Probabilities
  • Robustness to Useless Actions: Because the prior can assign low probability to actions that are rarely part of an optimal trajectory, the agent becomes much more efficient in environments with large, redundant action spaces.

Key Results

  • Superior Performance: On 19 tested Atari games, MIRL achieved a median normalized score 30% higher than standard Soft Q-Learning (SQL) and 50% higher than the classic Deep Q-Network (DQN).
  • Sample Efficiency: MIRL reached the final performance level of SQL in roughly half the number of environment interactions.
  • Winning the Majority: MIRL outperformed both baselines in 11 out of the 19 games tested. It showed particularly strong gains in BankHeist (nearly doubling the score of baselines) and Seaquest.

Limitations and Future Work

The authors identify one significant edge case: the "rare but vital" action. If a specific action is only useful in one tiny corner of the environment but is penalized or useless everywhere else, the learned prior might become so low that the agent "forgets" to try it when it actually matters. They observed hints of this limitation in their $8 \times 8$ grid world experiments.

Future directions include establishing formal convergence proofs for this alternating optimization (policy vs. prior) in non-tabular settings, and integrating MIRL into comprehensive frameworks like "Rainbow" to push the state-of-the-art even further.

Summary updated

Abstract

We propose a reinforcement learning (RL) algorithm that uses mutual-information regularization to optimize a prior action distribution for better performance and exploration. Entropy-based regularization has previously been shown to improve both exploration and robustness in challenging sequential decision-making tasks. It does so by encouraging policies to put probability mass on all actions. However, entropy regularization might be undesirable when actions have significantly different importance. In this paper, we propose a theoretically motivated framework that dynamically weights the importance of actions by using the mutual-information. In particular, we express the RL problem as an inference problem where the prior probability distribution over actions is subject to optimization. We show that the prior optimization introduces a mutual-information regularizer in the RL objective. This regularizer encourages the policy to be close to a non-uniform distribution that assigns higher probability mass to more important actions. We empirically demonstrate that our method significantly improves over entropy regularization methods and unregularized methods.
BibTeX
@inproceedings{graumoya2019soft,
  author = {Jordi Grau-Moya and Felix Leibfried and Peter Vrancx},
  title = {{Soft Q-Learning with Mutual-Information Regularization}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=LXH94-fdAKy},
}

Loading related content...

Figure