Reinforcement Learning with Augmented Data

Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, Aravind Srinivas

Abstract

Learning from visual observations is a fundamental yet challenging problem in Reinforcement Learning (RL). Although algorithmic advances combined with convolutional neural networks have proved to be a recipe for success, current methods are still lacking on two fronts: (a) data-efficiency of learning and (b) generalization to new environments. To this end, we present Reinforcement Learning with Augmented Data (RAD), a simple plug-and-play module that can enhance most RL algorithms. We perform the first extensive study of general data augmentations for RL on both pixel-based and state-based inputs, and introduce two new data augmentations - random translate and random amplitude scale. We show that augmentations such as random translate, crop, color jitter, patch cutout, random convolutions, and amplitude scale can enable simple RL algorithms to outperform complex state-of-the-art methods across common benchmarks. RAD sets a new state-of-the-art in terms of data-efficiency and final performance on the DeepMind Control Suite benchmark for pixel-based control as well as OpenAI Gym benchmark for state-based control. We further demonstrate that RAD significantly improves test-time generalization over existing methods on several OpenAI ProcGen benchmarks.

Introduction

Reinforcement Learning (RL) is notoriously data-hungry and prone to overfitting. While an agent might learn to master a specific task in a simulator, it often requires millions of frames to do so and fails immediately when faced with slight visual changes in a new environment. In Computer Vision (CV), Data Augmentation (DA)—the practice of creating "new" data by slightly modifying existing images—is a standard tool to prevent overfitting. However, in RL, this has been largely overlooked or relegated to complex auxiliary tasks. RAD addresses this by asking: Can we improve RL simply by applying standard data augmentations directly to the input observations without changing the underlying RL algorithm?

Methods

RAD is a simple module that can be wrapped around almost any RL algorithm, such as Soft Actor-Critic (SAC) for off-policy learning or Proximal Policy Optimization (PPO) for on-policy learning. The workflow involves sampling a minibatch of transitions, applying stochastic transformations to the observations, and training the agent on these augmented views.

Visualizing various data augmentations and the principle of consistent application across stacked frames to preserve temporal information.

A critical component of RAD is ensuring consistency across frame stacks. In RL, consecutive frames are often stacked to provide motion information. RAD applies the same random transformation across the entire stack to ensure temporal signals, like velocity, are not corrupted. Beyond standard CV techniques like cropping and flipping, the authors introduce Random Translate, which shifts the image within a larger frame, and Random Amplitude Scale, which multiplies state-based inputs by a random variable to improve robustness to signal magnitude changes.

Experimental Setup

The authors evaluated RAD across three diverse benchmark suites:

  1. DeepMind Control Suite (DMControl): For pixel-based continuous control tasks.
  2. OpenAI Gym: To test RAD on state-based (proprioceptive) inputs.
  3. OpenAI ProcGen: A suite of procedurally generated environments designed to test generalization to unseen levels.

Key Results and Insights

RAD achieves state-of-the-art results on all primary DMControl environments, providing a 4x efficiency boost over vanilla SAC and matching "oracle" agents that have access to internal simulator states. On the ProcGen benchmark, RAD significantly improves zero-shot performance; for instance, achieving a 55.8% relative gain in the BigFish environment.

Performance matrix showing the effectiveness of various augmentation combinations, highlighting the dominance of techniques like cropping.

One of the most surprising findings is that RAD (DA alone) outperforms complex algorithms like CURL, which uses contrastive learning losses. This suggests that the primary benefit of many recent representation learning papers in RL may stem from the data augmentations used rather than the auxiliary objectives themselves. For pixel-based tasks, Random Crop and Random Translate were the most effective, forcing the agent to learn translation invariance. Analysis of spatial attention maps shows that RAD helps agents ignore irrelevant background noise and focus on the robot's body.

Conclusion

While RAD is highly effective for visual generalization, it still faces challenges with structural generalization, such as entirely new map layouts in navigation tasks. Additionally, some augmentations may be "illegal" if they conflict with task-specific requirements (e.g., flipping in a game where direction is critical). Despite these limitations, RAD demonstrates that simple, well-applied data augmentation is a powerful tool for improving both the efficiency and robustness of RL agents.

Summary updated

Abstract

Learning from visual observations is a fundamental yet challenging problem in Reinforcement Learning (RL). Although algorithmic advances combined with convolutional neural networks have proved to be a recipe for success, current methods are still lacking on two fronts: (a) data-efficiency of learning and (b) generalization to new environments. To this end, we present Reinforcement Learning with Augmented Data (RAD), a simple plug-and-play module that can enhance most RL algorithms. We perform the first extensive study of general data augmentations for RL on both pixel-based and state-based inputs, and introduce two new data augmentations - random translate and random amplitude scale. We show that augmentations such as random translate, crop, color jitter, patch cutout, random convolutions, and amplitude scale can enable simple RL algorithms to outperform complex state-of-the-art methods across common benchmarks. RAD sets a new state-of-the-art in terms of data-efficiency and final performance on the DeepMind Control Suite benchmark for pixel-based control as well as OpenAI Gym benchmark for state-based control. We further demonstrate that RAD significantly improves test-time generalization over existing methods on several OpenAI ProcGen benchmarks. Our RAD module and training code are available at https://www.github.com/MishaLaskin/rad.
BibTeX
@inproceedings{laskin2020reinforcement,
  author = {Michael Laskin and Kimin Lee and Adam Stooke and Lerrel Pinto and Pieter Abbeel and Aravind Srinivas},
  title = {{Reinforcement Learning with Augmented Data}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=SMMmwSASMGq},
}
Figure