Introduction
Deep reinforcement learning (RL) agents are notorious for "memorizing" their environments. In classic benchmarks like the Arcade Learning Environment (Atari), agents often learn to execute highly specific sequences of actions tailored to fixed level layouts. If the environment changes even slightly, the agent's performance collapses.
While previous research has highlighted this overfitting problem, the community has lacked a comprehensive, standardized benchmark to measure RL generalization across a wide variety of tasks. To address this, researchers have introduced the Procgen Benchmark, a suite of 16 procedurally generated, game-like environments designed to evaluate sample efficiency and generalization in RL. By proving that RL agents easily overfit to fixed training levels, the authors demonstrate that highly diverse, randomized environments and larger neural network architectures are essential for training robust, generalizable agents.
The Procgen Benchmark
To force agents to generalize, the authors leverage Procedural Content Generation (PCG). Instead of human-designed, static levels, algorithms dynamically generate level layouts, asset selections, and enemy spawn locations every time an episode starts.
The Procgen Benchmark consists of 16 unique arcade-style environments (e.g., CoinRun, StarPilot, CaveFlyer, BossFight).
- Key Features: It uses a shared discrete action space (15 dimensions) and observation space (64x64x3 RGB). It features tunable difficulties (easy/hard) and is highly optimized for fast evaluation, capable of running thousands of steps per second on a single CPU core.
- Comparison to Existing Benchmarks: Unlike Atari, Procgen offers infinite level variations. Unlike earlier generalization tests, it provides a wide array of distinct games that are computationally cheap to run.
Experimental Approach
The authors establish standardized experimental protocols using Proximal Policy Optimization (PPO) and Rainbow DQN. They conduct several core experiments:
- Training Set Scaling: Training agents on discrete sets of levels (from 100 to 100,000) to see how much diversity is required to close the "generalization gap" (the difference between training and test performance).
- Deterministic Ablation: Training agents on a fixed sequence of levels (mimicking classic Atari setups) and testing them on randomized levels to expose the illusion of memorized progress.
- Model Scaling: Testing how the size of the neural network architecture impacts an agent's ability to handle highly diverse environments.
Key Insights and Results
The Illusion of Progress
When agents are trained on a fixed sequence of levels, their training scores skyrocket. However, when tested on unseen levels, their performance drops to near zero. They haven't learned to play the game; they've just memorized a specific obstacle course.
Massive Data Requirements for Generalization
Agents require access to a massive number of unique levels during training to fully close the generalization gap. As shown in the training set scaling experiments, it can take up to 10,000 unique levels for test performance to catch up to training performance.
The authors recommend a standard protocol of training on 500 levels and testing on the full distribution. This 500-level mark is the sweet spot where overfitting is highly visible, making it an excellent metric for evaluating future algorithmic improvements.
The "Implicit Curriculum" of Diversity
Surprisingly, in many Procgen environments, increasing the size of the training set eventually improves training performance. This runs counter to supervised learning norms, where more data usually makes fitting the training set harder. The authors hypothesize that highly diverse levels act as an implicit curriculum, forcing the agent to learn generalizable, robust skills that make mastering the entire training set easier.
Bigger Networks are Better for RL
Historically, RL has relied on relatively small neural networks. This paper reveals that when faced with highly diverse, procedurally generated environments, small networks (like the standard Nature-CNN) completely fail. Scaling the IMPALA-CNN parameter count by up to 16x yielded consistent, significant improvements in both training and test scores across the benchmark.
Algorithm Stability
PPO performed consistently well across the benchmark. Rainbow DQN, however, struggled with the high-variance gradients caused by the diverse environments, requiring heavy modifications just to train, and still underperformed PPO in consistency.
Limitations and Future Work
The authors acknowledge that Procgen environments are primarily designed to test visual recognition and motor control; by default, they require minimal memory or deep exploration. Additionally, the root cause of Rainbow DQN's instability on these environments remains an open question for future research.
Ultimately, the Procgen Benchmark serves as a robust, standardized proving ground, allowing researchers to leverage procedural generation to design more capable, efficient, and generalizable RL algorithms.
Summary updated