Regularization Matters in Policy Optimization

Zhuang Liu, Xuanlin Li, Bingyi Kang, Trevor Darrell

Abstract

Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention thanks to its encouraging performance on a variety of control tasks. Yet, conventional regularization techniques in training neural networks (e.g., $L_2$ regularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment, and because the deep RL community focuses more on high-level algorithm designs. In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks. Interestingly, we find conventional regularization techniques on the policy networks can often bring large improvement, especially on harder tasks. Our findings are shown to be robust against training hyperparameter variations. We also compare these techniques with the more widely used entropy regularization. In addition, we study regularizing different components and find that only regularizing the policy network is typically the best. We further analyze why regularization may help generalization in RL from four perspectives - sample complexity, reward distribution, weight norm, and noise robustness. We hope our study provides guidance for future practices in regularizing policy optimization algorithms.

Introduction

In supervised learning, regularization is a fundamental tool used to bridge the "generalization gap" between training and test data. However, in Deep Reinforcement Learning (RL), agents are typically trained and evaluated within the same environment. This lack of obvious distribution shift has led to a widespread assumption that overfitting is not a significant concern. Consequently, the research community has prioritized high-level algorithmic innovations, such as the transition from TRPO to PPO, while largely overlooking low-level network training "tricks" like weight decay or dropout.

This study challenges this status quo by conducting a systematic, large-scale empirical investigation into whether conventional regularization methods can improve performance in RL, even when training and testing environments are identical.

Methodology

The authors evaluate six distinct regularization techniques:

  1. $L_2$ Regularization (Weight Decay)
  2. $L_1$ Regularization
  3. Weight Clipping
  4. Dropout
  5. Batch Normalization (BN)
  6. Entropy Regularization (the current RL standard)

These techniques were integrated into four major policy optimization algorithms: A2C, TRPO, PPO (on-policy), and SAC (off-policy). The study also explores the architectural placement of regularization, specifically comparing its effects on the policy network (actor) versus the value network (critic).

The evaluation spans a diverse set of continuous control benchmarks, including MuJoCo (e.g., Ant, Humanoid) and the more challenging RoboSchool environments (e.g., AtlasForwardWalk). Discrete-action Atari environments were also included to verify the generalizability of the findings.

Key Insights: Where and What to Regularize

A critical finding of this research is that regularization should primarily be applied to the policy network. Applying these techniques to the value network often yields no benefit or even degrades performance. This is likely because the value network must track a non-stationary target (the TD error); over-constraining it with regularization may hinder its ability to adapt to rapid changes during training.

Among the techniques tested, $L_2$ regularization emerged as the most consistent performer. It frequently outperformed the standard entropy regularization across various algorithms and tasks. Conversely, Dropout and Batch Normalization were found to be detrimental to on-policy algorithms like PPO and TRPO. The authors suggest this is due to the noise introduced by these methods, which creates a discrepancy between the sampling policy and the optimization policy, leading to instability.

Key Results and Performance

The empirical results demonstrate that regularization significantly enhances both final performance and sample efficiency, particularly on complex tasks.

Training curves across various algorithms and environments show that $L_2$ and weight clipping often outperform the baseline and entropy regularization.
  • Performance Gains: $L_2$ regularization improved final performance in 52.8% of all tested environments, with the success rate rising to 66.7% on "hard" tasks. In some instances, such as PPO on the Humanoid task, the baseline failed to converge entirely, while the addition of $L_2$ regularization enabled the agent to find a high-performing solution.
  • Sample Efficiency: Regularized models consistently reached performance milestones using fewer environment interactions than their non-regularized counterparts.
Comparison of performance at different sample milestones, highlighting the sample efficiency gains from regularization.
  • Robustness: Beyond raw performance, agents trained with $L_2$ regularization exhibited greater robustness to input noise, suggesting that regularization helps the policy generalize better even within the same environment by smoothing the optimization landscape.

Conclusion

This work demonstrates that conventional regularization is a powerful yet underutilized tool in Deep RL. By keeping weight norms small and preventing premature policy certainty, techniques like $L_2$ weight decay can significantly improve the stability and efficiency of policy optimization. For practitioners, the message is clear: when facing difficult control tasks, a simple addition of $L_2$ regularization to the policy network may be more effective than switching to a more complex algorithm.

Code available at: https://github.com/xuanlinli17/iclr2021_rlreg

Summary updated

Abstract

Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention thanks to its encouraging performance on a variety of control tasks. Yet, conventional regularization techniques in training neural networks (e.g., $L_2$ regularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment, and because the deep RL community focuses more on high-level algorithm designs. In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks. Interestingly, we find conventional regularization techniques on the policy networks can often bring large improvement, especially on harder tasks. Our findings are shown to be robust against training hyperparameter variations. We also compare these techniques with the more widely used entropy regularization. In addition, we study regularizing different components and find that only regularizing the policy network is typically the best. We further analyze why regularization may help generalization in RL from four perspectives - sample complexity, reward distribution, weight norm, and noise robustness. We hope our study provides guidance for future practices in regularizing policy optimization algorithms. Our code is available at https://github.com/xuanlinli17/iclr2021_rlreg .
BibTeX
@inproceedings{liu2021regularization,
  author = {Zhuang Liu and Xuanlin Li and Bingyi Kang and Trevor Darrell},
  title = {{Regularization Matters in Policy Optimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=yr1mzrH3IC},
}

Loading related content...

Figure