Abstract: We introduce Procgen Benchmark, a suite of 16 procedurally generated game-like environments designed to benchmark both sample efficiency and generalization in reinforcement learning. We believe that the community will benefit from increased access to high quality training environments, and we provide detailed experimental protocols for using this benchmark. We empirically demonstrate that diverse environment distributions are essential to adequately train and evaluate RL agents, thereby motivating the extensive use of procedural content generation. We then use this benchmark to investigate the effects of scaling model size, finding that larger models significantly improve both sample efficiency and generalization.
Introduction: The Generalization Gap in RL
In Reinforcement Learning (RL), the Arcade Learning Environment (ALE)—comprising classic Atari games—has long served as the primary benchmark. However, ALE's static nature presents a significant limitation: agents can achieve high scores by memorizing specific sequences of actions rather than learning the underlying game logic. This leads to a substantial generalization gap, where agents fail when faced with even minor environmental variations. To address this, we propose the Procgen Benchmark, which emphasizes both sample efficiency and generalization by utilizing procedural content generation (PCG).
The Procgen Benchmark
The Procgen Benchmark consists of 16 diverse, game-like environments designed to be high-speed and accessible. Unlike traditional benchmarks, PCG algorithms generate level layouts, asset selections, and entity spawns dynamically. This provides a near-infinite supply of unique levels, forcing agents to learn robust, generalizable skills.
The suite includes various genres:
- Platformers/Shooters: CoinRun, StarPilot, Jumper, Climber, Ninja, BossFight.
- Maze/Navigation: CaveFlyer, Maze, Heist, Chaser, Miner.
- Classic-inspired: Leaper, Dodgeball, FruitBot, BigFish, Plunder.
Methodology
To evaluate generalization, we employ a "train/test" split methodology analogous to supervised learning. Agents are trained on a finite set of levels (e.g., 500) and subsequently evaluated on the "full distribution"—a stream of levels never encountered during training.
Our primary baseline algorithm is Proximal Policy Optimization (PPO), compared against Rainbow (a Q-learning variant). We also investigate the impact of model capacity by scaling the IMPALA-CNN architecture, specifically increasing the number of convolutional channels.
Key Insights and Results
The Impact of Environmental Diversity
Our experiments reveal that agents trained on a small, fixed set of levels often appear competent but fail entirely when tested on random levels. This confirms that agents frequently learn specific level seeds rather than game mechanics. We found that for many environments, the generalization gap only begins to close when agents are exposed to upwards of 10,000 unique training levels.
Interestingly, environmental diversity can act as an implicit curriculum. In several games, training on more levels actually improved training performance by preventing the agent from converging on local optima found in narrower datasets.
Scaling Laws and Algorithm Stability
We observed a significant correlation between model size and performance. Increasing the number of channels in the IMPALA-CNN by a factor of 4 (approximately a 16x increase in parameters) led to dramatic improvements in the Mean Normalized Score. Smaller architectures often failed to learn entirely when faced with high environmental diversity.
While Rainbow excelled in specific static contexts, PPO proved more stable across the diverse Procgen suite. Rainbow frequently suffered from gradient instability, likely due to the high variance inherent in procedurally generated worlds.
Conclusion
The Procgen Benchmark shifts the focus of RL evaluation from trajectory memorization to robust perception and generalizable logic. By providing highly optimized environments that run at thousands of steps per second, we enable researchers to conduct rigorous generalization studies with modest computational resources. Our findings suggest that both environmental diversity and model scale are fundamental pillars for developing agents that truly understand the worlds they inhabit.
Summary updated