Abstract
The NeurIPS 2020 Procgen Competition was designed as a centralized benchmark with clearly defined tasks for measuring Sample Efficiency and Generalization in Reinforcement Learning. Generalization remains one of the most fundamental challenges in deep reinforcement learning, and yet we do not have enough benchmarks to measure the progress of the community on Generalization in Reinforcement Learning. We present the design of a centralized benchmark for Reinforcement Learning which can help measure Sample Efficiency and Generalization in Reinforcement Learning by doing end to end evaluation of the training and rollout phases of thousands of user submitted code bases in a scalable way. We designed the benchmark on top of the already existing Procgen Benchmark by defining clear tasks and standardizing the end to end evaluation setups. The design aims to maximize the flexibility available for researchers who wish to design future iterations of such benchmarks, and yet imposes necessary practical constraints to allow for a system like this to scale. This paper presents the competition setup and the details and analysis of the top solutions identified through this setup in context of 2020 iteration of the competition at NeurIPS.
Introduction
In the field of Reinforcement Learning (RL), agents frequently suffer from "overfitting." An agent may achieve superhuman performance on a specific training level of a video game but fail to generalize when faced with minor environmental variations, such as a change in background color or a slight shift in platform placement. Traditional benchmarks, such as the Arcade Learning Environment (Atari), often allow agents to memorize specific action sequences rather than learning robust, generalizable concepts.
The NeurIPS 2020 Procgen Competition addressed this challenge by leveraging the Procgen Benchmark, a suite of 16 environments where every level is procedurally generated. This procedural generation forces agents to learn general policies that remain effective across an infinite variety of layouts and visual assets.
Competition Design and Tasks
The competition was structured to evaluate two critical dimensions of RL performance: Sample Efficiency and Generalization.
As illustrated in the figure above, the tracks differed primarily in the constraints placed on the training environment:
- Sample Efficiency Track: Agents were evaluated on their ability to learn a high-performing policy within a strict limit of 8 million environment steps, having access to all possible levels during training.
- Generalization Track: Agents were restricted to training on only 200 unique levels per environment. Their performance was then tested on a set of "held-out" levels they had never encountered, measuring how well the learned policy transferred to new instances of the same game mechanics.
Scalable Evaluation Pipeline
To handle the scale of the competition—which saw over 4,800 submissions—the organizers developed a centralized evaluation system. Participants submitted Docker images containing their training and rollout code. The system then orchestrated the training phase followed by a rollout phase to aggregate scores across multiple environments.
The benchmark utilized 16 diverse environments, including coinrun, bigfish, and miner. To ensure a rigorous test of "true" generalization, four specific games (caterpillar, gemjourney, hovercraft, and safezone) were kept secret from participants until the final evaluation.
Methodology of Top Solutions
The competition revealed several key technical strategies that separated top-performing agents from the baselines:
1. Algorithmic Shifts: From PPO to PPG
While Proximal Policy Optimization (PPO) served as the standard baseline, the winning teams transitioned toward Phasic Policy Gradient (PPG). PPG improves sample reuse by decoupling policy and value-function learning into distinct phases. This separation allows the agent to maintain a high-quality policy while more aggressively updating the value function, which proved beneficial for both efficiency and generalization.
2. The Critical Role of Data Augmentation
Data augmentation emerged as perhaps the most significant factor in achieving generalization. Techniques such as random translation, colored cutout, and "mixup" were employed to prevent neural networks from fixating on specific, non-essential visual features. The consensus among participants was that visual variety during training is often more impactful than the choice of the RL algorithm itself.
3. Architectural Enhancements
Many successful teams modified the standard IMPALA CNN architecture. A common finding was that the standard network was too "narrow" for the complexity of the Procgen tasks. Simply widening the convolutional layers provided a substantial performance boost by increasing the model's representational capacity.
4. Optimization and Stability
Given the strict 2-hour training limit on a single V100 GPU, teams utilized "one-cycle" learning rate schedules and reward normalization. These techniques helped stabilize training and ensured convergence within the allotted compute budget.
Key Results and Insights
The performance gap between the baseline and the top submissions was stark. In the Generalization track, the baseline PPO achieved a mean normalized reward of 0.2002, whereas the winning team (TRI) reached 0.6083. Similarly, in Sample Efficiency, the winner achieved a score of 0.7680 compared to the baseline's 0.3695.
Several "advanced" techniques were found to be counterproductive under the competition's constraints. Recurrent Neural Networks (RNNs) often slowed down training excessively, and certain auxiliary tasks (like contrastive learning) added computational overhead that outweighed their signal, leading to undertrained models.
Conclusion and Future Work
The NeurIPS 2020 Procgen Competition demonstrated that while standard algorithms are robust, significant gains in generalization and efficiency are achievable through better data augmentation, improved architectures like widened IMPALA, and refined algorithms like PPG.
Future iterations of such benchmarks should explore even more diverse procedural engines. Additionally, the community needs to address the "dull task" of hyperparameter tuning, which remains a significant bottleneck in researcher productivity. The success of this centralized benchmark provides a scalable template for future RL evaluations.
Summary updated