A Unified Framework for Regularized Reinforcement Learning

Xiang Li, Wenhao Yang, Zhihua Zhang

TL;DR

This paper introduces a general framework for regularized reinforcement learning (RL) that unifies existing entropy-based methods and allows for a broader class of regularizers. This framework enables the discovery of sparse and multi-modal optimal policies, which are more robust to unexpected situations and potentially more interpretable, while also providing theoretical guarantees and practical algorithms.

What Problem Does This Paper Address?

Traditional reinforcement learning aims to find an optimal policy that maximizes expected rewards. However, these optimal policies are often deterministic, meaning they select a single action for each state. This can be problematic in real-world scenarios where the chosen action might become unavailable or suboptimal due to unforeseen circumstances. Furthermore, deterministic policies can be less interpretable. While entropy-regularized RL methods offer a way to encourage stochasticity (multi-modality) and exploration, they often lead to policies that assign non-negligible probabilities to undesirable actions, potentially resulting in unsafe behavior. The authors address the need for RL methods that can find policies that are not only optimal but also exhibit desirable properties like sparsity (focusing on a few key actions) and multi-modality, leading to more robust and interpretable agents.

Methods

The core contribution is a unified framework for regularized Markov Decision Processes (MDPs). This framework extends the standard MDP objective by adding a general regularization term to the expected discounted total reward. This regularization term, denoted by $\lambda \phi(\pi(a|s))$, allows for a wide range of functions $\phi$ beyond just entropy.

Key aspects of their approach include:

  • General Regularization: They propose a framework where any function $\phi$ satisfying certain properties (monotonicity, non-negativity, differentiability, and expected concavity) can be used as a regularizer. This allows for exploring new types of regularizers that can induce sparsity and multi-modality.
  • Sparsity Conditions: They derive necessary and sufficient conditions for a regularization function to induce a sparse optimal policy, showing how the degree of sparsity can be controlled by the regularization coefficient $\lambda$.
  • Optimality Conditions: They provide a detailed mathematical analysis of the regularized MDPs, including deriving the optimality conditions for the regularized optimal policy and value functions.
  • Generic Method for Regularizers: They propose a method to combine existing regularizers (e.g., through addition or minimum) to create new ones.
  • Off-Policy Actor-Critic Algorithm (RAC): To handle complex environments, they develop a practical off-policy actor-critic algorithm that can optimize policies under this general regularization framework. This algorithm is designed to be agnostic to the specific form of the regularizer.

Datasets

The paper evaluates their proposed framework and algorithms on several environments:

  • Simple Random Generated MDP: A custom discrete environment with 50 states and 10 actions.
  • Gridworld Environment: A discrete environment with 81 states and 4 actions.
  • Atari Games: Standard benchmarks from OpenAI Gym (Alien, Boxing, Breakout, Seaquest).
  • MuJoCo Continuous Control Tasks: Standard benchmarks from OpenAI Gym (Ant-v2, Walker-v2, HalfCheetah-v2, Hopper-v2).

No new datasets were introduced.

Key Insights and Ideas

  • Regularization as a Control Knob: The choice of regularization function $\phi$ and its coefficient $\lambda$ acts as a powerful "knob" to control the properties of the optimal policy, such as its sparsity and multi-modality. This is a significant departure from relying solely on entropy.
  • Beyond Entropy: While entropy regularization is a special case, the framework reveals that many other functions (e.g., trigonometric, exponential) can induce sparsity and multi-modality, offering more flexibility.
  • Sparsity and Robustness: Sparse policies, by focusing on a subset of actions, can be more robust to unexpected situations. Multi-modal policies offer alternatives when the primary action fails. The framework allows for achieving both.
  • Theoretical Guarantees: The paper provides theoretical conditions for sparsity and analyzes the performance error introduced by regularization, offering guidance on choosing effective regularizers.
  • Unified View: The framework unifies various regularization approaches, providing a consistent theoretical foundation and algorithmic framework.

Key Results

  • Sparsity Control: The paper demonstrates that the regularization coefficient $\lambda$ can effectively control the sparsity of the optimal policy. As $\lambda$ increases, policies tend towards uniform distributions (less sparse), while smaller $\lambda$ values can lead to sparser policies.
  • Performance Comparison: Empirical results on discrete (random MDP, Gridworld) and continuous (Atari, MuJoCo) environments show that different regularizers exhibit varying performance.
  • In discrete environments, cosine regularization showed strong sparseness power.
  • In Atari games, Tsallis and exponential regularizers generally outperformed Shannon (entropy) regularization.
  • In MuJoCo tasks, Tsallis regularization was found to be more stable and less sensitive to $\lambda$ than Shannon and exponential regularizers.
  • Algorithm Effectiveness: The proposed Regularized Actor-Critic (RAC) algorithm successfully learns policies in complex environments using various regularizers.

Limitations and Future Work

  • Regularizer Choice: While the framework offers many choices, selecting the "best" regularizer for a specific task remains an empirical question. The paper provides insights but doesn't offer a definitive rule for all scenarios.
  • Numerical Stability: The paper notes that the cosine regularizer can be numerically unstable in deep RL training, potentially due to gradient exploding issues.
  • Hyperparameter Sensitivity: While some regularizers (like Tsallis) are shown to be less sensitive to $\lambda$, others (like Shannon) can be highly sensitive, requiring careful tuning.
  • Future Work: The authors suggest exploring more complex regularizers and investigating their application in even more challenging RL problems. They also highlight the potential for these sparse and multi-modal policies in real-world applications where robustness and interpretability are crucial.

Summary updated

Abstract

We propose and study a general framework for regularized Markov decision processes (MDPs) where the goal is to find an optimal policy that maximizes the expected discounted total reward plus a policy regularization term. The extant entropy-regularized MDPs can be cast into our framework. Moreover, under our framework, many regularization terms can bring multi-modality and sparsity, which are potentially useful in reinforcement learning. In particular, we present sufficient and necessary conditions that induce a sparse optimal policy. We also conduct a full mathematical analysis of the proposed regularized MDPs, including the optimality condition, performance error, and sparseness control. We provide a generic method to devise regularization forms and propose off-policy actor critic algorithms in complex environment settings. We empirically analyze the numerical properties of optimal policies and compare the performance of different sparse regularization forms in discrete and continuous environments.
BibTeX
@inproceedings{li2019unified,
  author = {Xiang Li and Wenhao Yang and Zhihua Zhang},
  title = {{A Unified Framework for Regularized Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=DCh2J79DmV},
}
Figure