Regularization Matters in Policy Optimization - An Empirical Study on Continuous Control

Zhuang Liu, Xuanlin Li, Bingyi Kang, Trevor Darrell

In the world of Supervised Learning (SL), techniques like $L_2$ regularization and dropout are the bread and butter of model training. You wouldn't dream of training a state-of-the-art computer vision model without them. Yet, in Deep Reinforcement Learning (RL), these "conventional" tools are strangely absent, often passed over in favor of high-level algorithmic tweaks or entropy regularization.

A team from UC Berkeley and the National University of Singapore recently challenged this status quo at ICLR 2021. Their paper, "Regularization Matters in Policy Optimization," provides a compelling empirical argument: we’ve been leaving performance on the table by ignoring the basics.

The Missing Generalization Gap

In supervised learning, we regularize to close the "generalization gap" between training and test data. In RL, however, agents are often trained and evaluated in the exact same environment. This has led to a prevailing—but perhaps misguided—belief that conventional regularization isn't necessary because there is no "gap" to bridge.

The authors argue that even within a single environment, there is a generalization gap between samples. An agent only sees a tiny fraction of possible trajectories during training. To perform well, it must generalize from those few seen samples to unseen parts of the vast state-action space. The researchers set out to see if the "old school" tricks of supervised learning could stabilize the notoriously finicky training process of Deep RL.

A Systematic Study of Regularization in RL

To test this, the researchers conducted the first comprehensive study of six regularization techniques across four major policy optimization algorithms:

  • Algorithms: PPO, TRPO, A2C (on-policy), and SAC (off-policy).
  • Regularizers: $L_2$ (weight decay), $L_1$, Weight Clipping, Dropout, Batch Normalization (BN), and the RL-standard Entropy Regularization.

They tested these across continuous control benchmarks (MuJoCo and RoboSchool), which are the gold standard for testing robotic locomotion and complex movement. Crucially, they experimented with where to apply the regularization: the policy network (the "actor"), the value network (the "critic"), or both.

Key Insights: Where and How to Regularize

1. The Policy Network is the Sweet Spot

The most consistent gains came from regularizing the policy network only. Interestingly, regularizing the value network often hurt performance. Because the value network's target is constantly shifting (non-stationary) during training, adding regularization makes it too hard for the critic to keep up and accurately estimate values.

2. $L_2$ is the MVP

Across almost all algorithms and tasks, $L_2$ regularization was the most effective. It often outperformed the more common entropy regularization, especially on difficult, high-dimensional tasks like the Humanoid.

Learning curves across various continuous control tasks demonstrating the performance boost of L2 and L1 regularization over baselines.

3. On-Policy vs. Off-Policy Friction

Batch Normalization and Dropout—standard in SL—actually crippled on-policy algorithms like PPO and TRPO. Why? These algorithms rely heavily on the data being "on-policy." The stochasticity introduced by BN and Dropout creates a discrepancy between the policy used to collect data and the policy being optimized, violating the core theory of these methods. However, they worked quite well for the off-policy SAC algorithm.

Why it Works: Mechanisms of Improvement

Beyond just achieving higher scores, the study revealed why regularized agents perform better:

  • Sample Complexity: Regularization significantly reduced sample complexity. Regularized agents reached the same performance levels as the baseline using far fewer environment interactions.
  • Stability: Regularized models showed much tighter return distributions. While a baseline agent might have some "lucky" high-return runs and many "failed" low-return runs, regularized agents were consistently successful.
  • Weight Norms & Robustness: The authors found that $L_2$ successfully kept the policy network's weight norms small. This smaller norm directly correlated with better robustness to noise in the input observations, preventing the network from overfitting to specific, noisy states.
Correlation between return, policy weight norm, and policy entropy. L2 regularization keeps the weight norm low, correlating with higher returns.

The Takeaway

While the study focuses heavily on policy optimization and leaves the theoretical "why" of value network degradation open for future work, the practical takeaway for RL practitioners is crystal clear.

If you’re training a policy network on a tough continuous control task, turn on $L_2$ regularization for the actor. It’s a simple, computationally cheap change that stabilizes training, improves sample efficiency, and boosts overall performance far more than conventional RL wisdom previously suggested.

Summary updated

Abstract

Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention thanks to its encouraging performance on a variety of control tasks. Yet, conventional regularization techniques in training neural networks (e.g., $L_2$ regularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment, and because the deep RL community focuses more on high-level algorithm designs. In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks. Interestingly, we find conventional regularization techniques on the policy networks can often bring large improvement, especially on harder tasks. Our findings are shown to be robust against training hyperparameter variations. We also compare these techniques with the more widely used entropy regularization. In addition, we study regularizing different components and find that only regularizing the policy network is typically the best. We further analyze why regularization may help generalization in RL from four perspectives - sample complexity, reward distribution, weight norm, and noise robustness. We hope our study provides guidance for future practices in regularizing policy optimization algorithms. Our code is available at https://github.com/xuanlinli17/iclr2021_rlreg .
BibTeX
@inproceedings{liu2021regularization,
  author = {Zhuang Liu and Xuanlin Li and Bingyi Kang and Trevor Darrell},
  title = {{Regularization Matters in Policy Optimization - An Empirical Study on Continuous Control}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=yr1mzrH3IC},
}

Loading related content...

Figure