In the world of Supervised Learning (SL), techniques like $L_2$ regularization and dropout are the bread and butter of model training. You wouldn't dream of training a state-of-the-art computer vision model without them. Yet, in Deep Reinforcement Learning (RL), these "conventional" tools are strangely absent, often passed over in favor of high-level algorithmic tweaks or entropy regularization.
A team from UC Berkeley and the National University of Singapore recently challenged this status quo at ICLR 2021. Their paper, "Regularization Matters in Policy Optimization," provides a compelling empirical argument: we’ve been leaving performance on the table by ignoring the basics.
The Missing Generalization Gap
In supervised learning, we regularize to close the "generalization gap" between training and test data. In RL, however, agents are often trained and evaluated in the exact same environment. This has led to a prevailing—but perhaps misguided—belief that conventional regularization isn't necessary because there is no "gap" to bridge.
The authors argue that even within a single environment, there is a generalization gap between samples. An agent only sees a tiny fraction of possible trajectories during training. To perform well, it must generalize from those few seen samples to unseen parts of the vast state-action space. The researchers set out to see if the "old school" tricks of supervised learning could stabilize the notoriously finicky training process of Deep RL.
A Systematic Study of Regularization in RL
To test this, the researchers conducted the first comprehensive study of six regularization techniques across four major policy optimization algorithms:
- Algorithms: PPO, TRPO, A2C (on-policy), and SAC (off-policy).
- Regularizers: $L_2$ (weight decay), $L_1$, Weight Clipping, Dropout, Batch Normalization (BN), and the RL-standard Entropy Regularization.
They tested these across continuous control benchmarks (MuJoCo and RoboSchool), which are the gold standard for testing robotic locomotion and complex movement. Crucially, they experimented with where to apply the regularization: the policy network (the "actor"), the value network (the "critic"), or both.
Key Insights: Where and How to Regularize
1. The Policy Network is the Sweet Spot
The most consistent gains came from regularizing the policy network only. Interestingly, regularizing the value network often hurt performance. Because the value network's target is constantly shifting (non-stationary) during training, adding regularization makes it too hard for the critic to keep up and accurately estimate values.
2. $L_2$ is the MVP
Across almost all algorithms and tasks, $L_2$ regularization was the most effective. It often outperformed the more common entropy regularization, especially on difficult, high-dimensional tasks like the Humanoid.
3. On-Policy vs. Off-Policy Friction
Batch Normalization and Dropout—standard in SL—actually crippled on-policy algorithms like PPO and TRPO. Why? These algorithms rely heavily on the data being "on-policy." The stochasticity introduced by BN and Dropout creates a discrepancy between the policy used to collect data and the policy being optimized, violating the core theory of these methods. However, they worked quite well for the off-policy SAC algorithm.
Why it Works: Mechanisms of Improvement
Beyond just achieving higher scores, the study revealed why regularized agents perform better:
- Sample Complexity: Regularization significantly reduced sample complexity. Regularized agents reached the same performance levels as the baseline using far fewer environment interactions.
- Stability: Regularized models showed much tighter return distributions. While a baseline agent might have some "lucky" high-return runs and many "failed" low-return runs, regularized agents were consistently successful.
- Weight Norms & Robustness: The authors found that $L_2$ successfully kept the policy network's weight norms small. This smaller norm directly correlated with better robustness to noise in the input observations, preventing the network from overfitting to specific, noisy states.
The Takeaway
While the study focuses heavily on policy optimization and leaves the theoretical "why" of value network degradation open for future work, the practical takeaway for RL practitioners is crystal clear.
If you’re training a policy network on a tough continuous control task, turn on $L_2$ regularization for the actor. It’s a simple, computationally cheap change that stabilizes training, improves sample efficiency, and boosts overall performance far more than conventional RL wisdom previously suggested.
Summary updated