TL;DR
The NeurIPS 2020 Procgen Competition established a rigorous framework for measuring how well Reinforcement Learning (RL) agents generalize to unseen environments while remaining sample-efficient. By leveraging the procedurally generated "Procgen" benchmark, the competition revealed that Phasic Policy Gradient (PPG) and heavy data augmentation are currently the most effective tools for preventing RL agents from simply "memorizing" their training data.
What Problem Does This Paper Address?
Deep Reinforcement Learning has seen massive success in games like Go and Atari, but these agents often suffer from extreme overfitting. An agent might become a master at a specific level of a game but fail completely if the colors change or platforms shift by a few pixels. This lack of generalization is a major hurdle for real-world applications.
Furthermore, many RL benchmarks allow for nearly infinite training steps, which doesn't reflect real-world constraints. This paper summarizes a competition designed to tackle these two issues simultaneously:
- Sample Efficiency: Can an agent reach high performance within a strict budget of 8 million environment steps?
- Generalization: Can an agent trained on a limited set of levels perform well on entirely new, procedurally generated levels?
Methods
The competition was hosted on the AIcrowd platform and used a "code-submission" model, where participants submitted their training and rollout code rather than pre-trained models. This ensured that all agents were trained from scratch under identical constraints (e.g., a single NVIDIA V100 GPU and a 2-hour training limit).
The top-performing teams primarily used variations of two algorithms:
- Phasic Policy Gradient (PPG): An extension of PPO that separates policy and value-function learning into distinct phases, allowing for higher sample reuse without destabilizing the policy.
- Proximal Policy Optimization (PPO): The industry standard, often modified with architectural tweaks.
The "Secret Sauce" for Success: * Data Augmentation: Techniques like "random translate" (shifting the image slightly) were crucial. They forced the agents to learn features rather than specific pixel locations. * Architectural Scaling: Many winners replaced the standard IMPALA CNN with wider networks, finding that increased model capacity significantly improved generalization. * Reward Normalization: Scaling rewards to a consistent range helped stabilize learning across diverse environments.
Datasets
The competition utilized the Procgen Benchmark, which consists of 16 diverse, procedurally generated environments (e.g., CoinRun, StarPilot, BigFish).
- Training Set: Participants had access to 16 public environments. In the generalization track, agents were restricted to seeing only 200 unique levels per environment during training.
- Test Set: Evaluation was performed on 4 "hold-out" environments (Caterpillar, GemJourney, Hovercraft, Safezone) that were never released to the participants, providing a true test of out-of-distribution generalization.
Key Insights and Ideas
- Augmentation is a Double-Edged Sword: While data augmentation is vital for generalization, the authors found that keeping a percentage of frames un-augmented was necessary to maintain policy stability.
- PPG > PPO: The top two teams in the generalization category used PPG. Its ability to decouple the training of the value function from the policy allows for more aggressive learning without the "forgetting" issues common in standard PPO.
- Capacity Matters: Simply making the neural network wider (increasing the number of channels in the IMPALA encoder) led to drastic performance gains, suggesting that RL agents are often under-parameterized for complex, procedural tasks.
- Auxiliary Tasks can be Distractions: Interestingly, complex auxiliary tasks like contrastive learning or DeepMDP actually degraded performance. The overhead of these methods reduced the total number of training steps possible within the time limit, leading to undertrained models.
Key Results
The competition saw a massive leap over the baseline PPO implementation provided by the organizers:
- Generalization Track: The winning team (TRI) achieved a mean normalized reward of 0.6083, compared to the baseline's 0.2002.
- Sample Efficiency Track: Team TRI also won here with a score of 0.7680, more than doubling the baseline's 0.3695.
- Diversity of Performance: While agents performed consistently well on games like Fruitbot, they struggled significantly with Leaper and Dodgeball, indicating that certain types of logic (like complex timing or multi-object tracking) remain difficult for current RL architectures to generalize.
Limitations and Future Work
The authors acknowledge that hyperparameter tuning remains a "dull" but dominant factor in success. Most top teams spent significant effort on grid searches, which is computationally expensive and hard to replicate in resource-constrained settings.
Future work suggests a need for:
- More automated and efficient hyperparameter optimization specifically for RL.
- Investigation into why recurrent neural networks (RNNs) performed poorly (they were too slow for the 8M step limit), as memory is crucial for many real-world tasks.
- Developing methods that can incorporate auxiliary tasks without the heavy computational overhead that hindered participants in this competition.
Summary updated