Introduction
Deep Reinforcement Learning (RL) has achieved incredible feats, from superhuman Atari gameplay to controlling complex robots. However, learning directly from visual inputs is notoriously plagued by two major issues: data inefficiency (requiring millions of frames to learn simple tasks) and poor generalization (agents often fail completely if background colors or layouts change slightly).
In supervised computer vision, these exact problems were largely mitigated years ago using data augmentation (e.g., randomly cropping or flipping training images). Surprisingly, RL has largely bypassed simple data augmentation, instead relying on highly complex workarounds like building predictive world models or using auxiliary contrastive losses (such as the CURL algorithm). This paper asks a straightforward question: Can we just augment the input data and feed it directly to a standard RL algorithm?
The result is RAD (Reinforcement Learning with Augmented Data), a simple, plug-and-play module that applies standard computer vision data augmentations directly to RL inputs, achieving state-of-the-art data efficiency and generalization without changing underlying RL algorithms.
Methods: The RAD Module
RAD acts as a module sitting between the environment and the RL agent. It intercepts observations (images or state vectors), applies stochastic augmentations, and passes them to the agent. Because it only modifies the input stream, RAD is entirely plug-and-play and requires zero changes to the underlying RL algorithm (tested with SAC for continuous control and PPO for discrete control).
A crucial component of RAD is the frame-stacking trick. In RL, agents usually receive a "stack" of consecutive frames to infer velocity and momentum. RAD applies augmentations randomly across a batch of data, but consistently across the frames within a single stack. Without this consistency, the agent would perceive the world as violently shaking, destroying vital temporal information.
The authors tested a wide array of image augmentations, including crop, grayscale, cutout, flip, rotate, random convolution, and color-jitter. They also introduced two novel visual augmentations: random translate and random window. Furthermore, RAD is not limited to pixels; for proprioceptive, state-based inputs (e.g., joint positions), they introduce random amplitude scaling and Gaussian noise.
Key Insights and Ideas
- Simplicity Beats Complexity: RAD proves that auxiliary representation-learning losses or complex generative world models are not strictly necessary to achieve state-of-the-art RL from pixels. Simple, well-applied data augmentation is sufficient.
- Translation is the Secret Sauce of Cropping: The authors found that "random crop" was the most effective image augmentation. By ablating the cropping operation, they discovered it performs two distinct functions: hiding information at the edges (windowing) and shifting the image (translation). The information hiding provided almost no benefit; the performance boost came almost entirely from forcing the agent to be invariant to spatial translation.
- Augmentation Fixes the Agent's Attention: Visualizing the neural network's spatial attention maps revealed that baseline agents without augmentation overfit to specific, irrelevant background pixels. RAD—specifically with cropping and translation—forces the agent's attention directly onto the robot's body, creating a robust representation of the actual state.
- Augmentation Works for Raw State Vectors: Applying "random amplitude scaling" to non-visual state arrays forces the agent to rely on the intrinsic relationships and signs of the inputs rather than absolute magnitudes, drastically improving standard state-based RL.
Evaluation and Results
The authors evaluated RAD across three distinct suites to test different capabilities:
- DeepMind Control Suite (Data Efficiency): RAD achieved state-of-the-art performance on all 6 benchmarked continuous control environments at both 100k and 500k steps, beating complex baselines like CURL, Dreamer, and SLAC. It improved the performance of baseline pixel-SAC by 4x. Amazingly, pixel-based RAD matched the performance of an "oracle" state-based agent on 11 out of 15 environments.
- OpenAI ProcGen (Zero-Shot Generalization): Agents were trained on a set of procedural levels and tested on completely unseen levels. RAD with random crop achieved a 55.8% relative gain over baseline PPO on the BigFish environment. Furthermore, an agent trained with RAD on just 100 levels outperformed a baseline agent trained on 200 levels, proving augmentation is more efficient than simply collecting more diverse data.
- OpenAI Gym (State-Based Control): Using random amplitude scaling, RAD achieved SOTA on 4 out of 6 environments, outperforming complex model-based methods (e.g., achieving 1.7x the return of POPLIN-P on the Walker task).
Limitations and Future Work
While highly effective for visual and state variations, data augmentation was less effective on tasks requiring structural generalization, such as the ProcGen Jumper environment. Because Jumper is a navigation task, it requires adapting to entirely new map layouts. Visual augmentations do not help much here; solving this likely requires recurrent policies (RNNs) to remember map layouts.
Looking forward, RAD's low compute requirements and high efficiency make it a prime candidate for real-world robotics, where data collection is expensive and environments are visually noisy. Further study into augmentations tailored for environments with drastic visual theme changes could yield even more robust RL agents.
Summary updated