Introduction
In standard Reinforcement Learning (RL), the objective is typically to find a deterministic optimal policy—one "best" action per state. However, deterministic policies are inherently brittle; if the preferred action becomes unavailable or fails, the agent lacks a backup plan.
To address this, researchers often employ Shannon entropy regularization, which yields stochastic, "Softmax" policies. While these policies are multi-modal and robust, they suffer from being "dense": they assign a non-zero probability to every possible action, including highly suboptimal or dangerous ones. In high-dimensional action spaces, this wastes probability mass and can be unsafe.
This paper introduces a unified mathematical framework for regularizing RL policies using a wide array of functions beyond standard entropy. By identifying the specific mathematical conditions that lead to "sparse" policies—those that assign exactly zero probability to suboptimal actions—the authors enable agents to remain multi-modal and robust without entertaining terrible choices.
The Regularized MDP Framework
The authors propose a general framework for Regularized Markov Decision Processes (MDPs). Instead of strictly maximizing expected rewards, the agent maximizes the reward plus a regularization term, $\lambda \phi(\pi)$.
Key methodological contributions include:
- Regularization Assumptions: The authors define four conditions (Monotonicity, Non-negativity, Differentiability, and Expected Concavity) that a function must satisfy to serve as a valid regularizer.
- The Sparsity Condition: A critical theoretical threshold is proven: a regularizer induces a sparse policy if and only if its derivative at zero is finite. This explains why Shannon entropy (whose derivative at zero is infinite) can never produce a truly sparse policy.
- Novel Regularizers: Moving beyond Shannon and Tsallis entropy, the paper introduces new "basic" regularizers based on Cosine and Exponential functions. It also provides methods to compose them, such as taking the minimum or sum of different regularizers to create hybrid behaviors.
- Regularized Actor-Critic (RAC): To apply these theories, the authors design a generic, off-policy actor-critic algorithm that is "regularizer-agnostic." It utilizes twin Q-networks to mitigate overestimation bias and adapts to both discrete and continuous action spaces.
Key Insights and Theoretical Properties
The framework reveals several important properties of regularized policies:
- The "Sparsity Knob": The regularization coefficient $\lambda$ acts as a direct control for the degree of sparsity. As $\lambda \to 0$, the policy collapses to a single deterministic action. As $\lambda \to \infty$, the policy approaches a uniform distribution.
- Action Filtration: Sparse regularizers implicitly perform "action filtration." By zeroing out suboptimal actions entirely, the agent focuses its learning capacity only on the "near-optimal" candidates.
- Tighter Error Bounds: Theoretical and empirical analyses show that sparse regularizers (like Tsallis or the proposed Cosine form) often exhibit a smaller "performance error" relative to the true optimal value compared to Shannon entropy.
Empirical Results
The authors evaluated their framework across numerical simulations, discrete control tasks (Atari 2600), and continuous control benchmarks (MuJoCo).
Discrete Environments (Atari): In complex discrete environments, the Cosine and Tsallis regularizers frequently outperformed standard Shannon entropy. For instance, the Cosine regularizer achieved the highest scores in games like Alien and Seaquest, while Tsallis led in Boxing.
Continuous Environments (MuJoCo): In continuous control tasks (Ant-v2, Walker-v2, Hopper-v2, HalfCheetah-v2), Tsallis regularization proved consistently more stable and achieved higher rewards than Shannon entropy. The variance during training was significantly lower for Tsallis, indicating more reliable convergence.
Limitations and Future Directions
While the framework is highly flexible, it introduces new challenges. The Cosine regularizer, despite its success in discrete tasks, proved unstable in continuous MuJoCo environments. The authors attribute this to the periodicity of the cosine function, which can cause gradients to vacillate and create numerical instabilities where probability densities diverge. Additionally, the choice of the regularization coefficient $\lambda$ and the specific form of $\phi$ still require careful hyperparameter tuning for different environments.
Future work will likely explore more complex regularization forms and investigate how these sparse policies adapt in non-stationary environments where the set of viable actions changes dynamically over time.
Summary updated