Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmark

Sharada P. Mohanty, Jyotish Poonganam, Adrien Gaidon, Andrey Kolobov, Blake Wulfe, Dipam Chakraborty, Grazvydas Semetulskis, João Schapke, Jonas Kubilius, Jurgis Pasukonis, Linas Klimas, Matthew J. Hausknecht, Patrick MacAlpine, Quang Nhat Tran, Thomas Tumiel, Xiaocheng Tang, Xinwei Chen, Christopher Hesse, Jacob Hilton, William Hebgen Guss, Sahika Genc, John Schulman, Karl Cobbe

Abstract

The NeurIPS 2020 Procgen Competition was designed as a centralized benchmark with clearly defined tasks for measuring Sample Efficiency and Generalization in Reinforcement Learning. Generalization remains one of the most fundamental challenges in deep reinforcement learning, and yet we do not have enough benchmarks to measure the progress of the community on Generalization in Reinforcement Learning. We present the design of a centralized benchmark for Reinforcement Learning which can help measure Sample Efficiency and Generalization in Reinforcement Learning by doing end to end evaluation of the training and rollout phases of thousands of user submitted code bases in a scalable way. We designed the benchmark on top of the already existing Procgen Benchmark by defining clear tasks and standardizing the end to end evaluation setups. The design aims to maximize the flexibility available for researchers who wish to design future iterations of such benchmarks, and yet imposes necessary practical constraints to allow for a system like this to scale. This paper presents the competition setup and the details and analysis of the top solutions identified through this setup in context of 2020 iteration of the competition at NeurIPS.

Introduction

In the field of Reinforcement Learning (RL), agents frequently suffer from "overfitting." An agent may achieve superhuman performance on a specific training level of a video game but fail to generalize when faced with minor environmental variations, such as a change in background color or a slight shift in platform placement. Traditional benchmarks, such as the Arcade Learning Environment (Atari), often allow agents to memorize specific action sequences rather than learning robust, generalizable concepts.

The NeurIPS 2020 Procgen Competition addressed this challenge by leveraging the Procgen Benchmark, a suite of 16 environments where every level is procedurally generated. This procedural generation forces agents to learn general policies that remain effective across an infinite variety of layouts and visual assets.

Competition Design and Tasks

The competition was structured to evaluate two critical dimensions of RL performance: Sample Efficiency and Generalization.

Comparison of the Sample Efficiency and Generalization evaluation tracks.

As illustrated in the figure above, the tracks differed primarily in the constraints placed on the training environment:

  • Sample Efficiency Track: Agents were evaluated on their ability to learn a high-performing policy within a strict limit of 8 million environment steps, having access to all possible levels during training.
  • Generalization Track: Agents were restricted to training on only 200 unique levels per environment. Their performance was then tested on a set of "held-out" levels they had never encountered, measuring how well the learned policy transferred to new instances of the same game mechanics.

Scalable Evaluation Pipeline

To handle the scale of the competition—which saw over 4,800 submissions—the organizers developed a centralized evaluation system. Participants submitted Docker images containing their training and rollout code. The system then orchestrated the training phase followed by a rollout phase to aggregate scores across multiple environments.

The centralized evaluation pipeline for multiple Procgen environments.

The benchmark utilized 16 diverse environments, including coinrun, bigfish, and miner. To ensure a rigorous test of "true" generalization, four specific games (caterpillar, gemjourney, hovercraft, and safezone) were kept secret from participants until the final evaluation.

Methodology of Top Solutions

The competition revealed several key technical strategies that separated top-performing agents from the baselines:

1. Algorithmic Shifts: From PPO to PPG

While Proximal Policy Optimization (PPO) served as the standard baseline, the winning teams transitioned toward Phasic Policy Gradient (PPG). PPG improves sample reuse by decoupling policy and value-function learning into distinct phases. This separation allows the agent to maintain a high-quality policy while more aggressively updating the value function, which proved beneficial for both efficiency and generalization.

2. The Critical Role of Data Augmentation

Data augmentation emerged as perhaps the most significant factor in achieving generalization. Techniques such as random translation, colored cutout, and "mixup" were employed to prevent neural networks from fixating on specific, non-essential visual features. The consensus among participants was that visual variety during training is often more impactful than the choice of the RL algorithm itself.

3. Architectural Enhancements

Many successful teams modified the standard IMPALA CNN architecture. A common finding was that the standard network was too "narrow" for the complexity of the Procgen tasks. Simply widening the convolutional layers provided a substantial performance boost by increasing the model's representational capacity.

4. Optimization and Stability

Given the strict 2-hour training limit on a single V100 GPU, teams utilized "one-cycle" learning rate schedules and reward normalization. These techniques helped stabilize training and ensured convergence within the allotted compute budget.

Key Results and Insights

The performance gap between the baseline and the top submissions was stark. In the Generalization track, the baseline PPO achieved a mean normalized reward of 0.2002, whereas the winning team (TRI) reached 0.6083. Similarly, in Sample Efficiency, the winner achieved a score of 0.7680 compared to the baseline's 0.3695.

Several "advanced" techniques were found to be counterproductive under the competition's constraints. Recurrent Neural Networks (RNNs) often slowed down training excessively, and certain auxiliary tasks (like contrastive learning) added computational overhead that outweighed their signal, leading to undertrained models.

Conclusion and Future Work

The NeurIPS 2020 Procgen Competition demonstrated that while standard algorithms are robust, significant gains in generalization and efficiency are achievable through better data augmentation, improved architectures like widened IMPALA, and refined algorithms like PPG.

Future iterations of such benchmarks should explore even more diverse procedural engines. Additionally, the community needs to address the "dull task" of hyperparameter tuning, which remains a significant bottleneck in researcher productivity. The success of this centralized benchmark provides a scalable template for future RL evaluations.

Summary updated

Abstract

The NeurIPS 2020 Procgen Competition was designed as a centralized benchmark with clearly defined tasks for measuring Sample Efficiency and Generalization in Reinforcement Learning. Generalization remains one of the most fundamental challenges in deep reinforcement learning, and yet we do not have enough benchmarks to measure the progress of the community on Generalization in Reinforcement Learning. We present the design of a centralized benchmark for Reinforcement Learning which can help measure Sample Efficiency and Generalization in Reinforcement Learning by doing end to end evaluation of the training and rollout phases of thousands of user submitted code bases in a scalable way. We designed the benchmark on top of the already existing Procgen Benchmark by defining clear tasks and standardizing the end to end evaluation setups. The design aims to maximize the flexibility available for researchers who wish to design future iterations of such benchmarks, and yet imposes necessary practical constraints to allow for a system like this to scale. This paper presents the competition setup and the details and analysis of the top solutions identified through this setup in context of 2020 iteration of the competition at NeurIPS.
BibTeX
@inproceedings{mohanty2020measuring,
  author = {Sharada P. Mohanty and Jyotish Poonganam and Adrien Gaidon and Andrey Kolobov and Blake Wulfe and Dipam Chakraborty and Grazvydas Semetulskis and João Schapke and Jonas Kubilius and Jurgis Pasukonis and Linas Klimas and Matthew J. Hausknecht and Patrick MacAlpine and Quang Nhat Tran and Thomas Tumiel and Xiaocheng Tang and Xinwei Chen and Christopher Hesse and Jacob Hilton and William Hebgen Guss and Sahika Genc and John Schulman and Karl Cobbe},
  title = {{Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmark}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=YPA0CVM-aQd},
}
Figure