A Survey of Generalisation in Deep Reinforcement Learning

Robert Kirk, Amy Zhang, Edward Grefenstette, Tim Rocktäschel

Abstract

The study of zero-shot generalisation (ZSG) in deep Reinforcement Learning (RL) aims to produce RL algorithms whose policies generalise well to novel unseen situations at deployment time, avoiding overfitting to their training environments. Tackling this is vital if we are to deploy reinforcement learning algorithms in real world scenarios, where the environment will be diverse, dynamic and unpredictable. This survey is an overview of this nascent field. We rely on a unifying formalism and terminology for discussing different ZSG problems, building upon previous works. We go on to categorise existing benchmarks for ZSG, as well as current methods for tackling these problems. Finally, we provide a critical discussion of the current state of the field, including recommendations for future work. Among other conclusions, we argue that taking a purely procedural content generation approach to benchmark design is not conducive to progress in ZSG, we suggest fast online adaptation and tackling RL-specific problems as some areas for future work on methods for ZSG, and we recommend building benchmarks in underexplored problem settings such as offline RL ZSG and reward-function variation.

Introduction: The Problem of Evaluation Overfitting

In classical Reinforcement Learning (RL), success is often measured by an agent's ability to master a specific task, such as a single level in a video game or a specific robotic simulation. However, these agents are notoriously brittle; minor changes to the environment—such as background color shifts or slight adjustments in object placement—can cause policy performance to collapse. This phenomenon is known as "evaluation overfitting."

To move toward real-world reliability, RL agents must exhibit Zero-shot Generalisation (ZSG): the ability to perform well in novel, unseen situations at deployment time without additional training. This survey addresses the lack of a common language and clear taxonomy in this vital sub-field, providing a roadmap for future research.

Formalizing Generalisation in RL

To unify the field, the authors utilize a formalism based on Contextual Markov Decision Processes (CMDPs). In a CMDP, the environment is defined by a "context" ($c$), representing hidden parameters like gravity, friction, or layout. By separating the agent's state from the environment's context, we can formally define the transfer of a policy from a training distribution of contexts to a testing distribution.

Comparison of Singleton, IID, and OOD environments across graphical models, distributions, and benchmarks.

As illustrated in the figure above, the field distinguishes between three primary settings:

  1. Singleton Environments: Standard MDPs where training and testing occur in the same environment.
  2. IID Generalisation: Training and testing contexts are drawn from the same distribution ($p_{train}(c) = p_{test}(c)$).
  3. OOD Generalisation: The testing distribution differs from the training distribution ($p_{train}(c) \neq p_{test}(c)$), representing the most challenging and realistic scenario.

Taxonomy of Methods

Existing ZSG methods are categorized into three main pillars:

  • Increasing Similarity: Techniques like Data Augmentation, Domain Randomization, and Procedural Environment Generation aim to make the training distribution broad enough to cover potential test scenarios.
  • Handling Differences: Methods that explicitly manage the gap between environments, such as encoding Inductive Biases, using Regularization to prevent memorization, and Fast Online Adaptation.
  • RL-Specific Improvements: Addressing unique RL challenges like non-stationarity and the exploration-exploitation trade-off during the learning process.

Benchmarks and Key Insights

The survey catalogs 55 different environments, distinguishing between Procedural Content Generation (PCG) (e.g., OpenAI Procgen) and Controllable Environments (e.g., CausalWorld). While PCG is popular, the authors argue it can be a "trap" because random seeds make it difficult to isolate which specific factors (visuals vs. physics) cause failure.

Key findings from the survey include:

  • Visual Variation is Over-represented: 53% of environments focus on observation changes (colors/textures).
  • Dynamics Variation is "Hard Mode": Generalizing to different physics (gravity, friction) remains a major hurdle.
  • Inductive Biases are Mandatory: Extrapolating to truly unseen environments requires the agent's architecture to have prior knowledge about the world (e.g., object permanence or local physics).

Future Directions

The authors identify several critical "blind spots" for the field:

  1. Offline RL ZSG: Developing benchmarks for generalizing from static, pre-collected datasets rather than active interaction.
  2. Context Efficiency: Measuring how many training environments are needed to achieve generalisation.
  3. Reward-Function Variation: Building agents that can handle multi-objective or underspecified rewards, which is essential for human-AI interaction.
  4. Fast Online Adaptation: Moving toward agents that can "feel out" new environment parameters within a single episode using recurrent policies.

Summary updated

Abstract

The study of zero-shot generalisation (ZSG) in deep Reinforcement Learning (RL) aims to produce RL algorithms whose policies generalise well to novel unseen situations at deployment time, avoiding overfitting to their training environments. Tackling this is vital if we are to deploy reinforcement learning algorithms in real world scenarios, where the environment will be diverse, dynamic and unpredictable. This survey is an overview of this nascent field. We rely on a unifying formalism and terminology for discussing different ZSG problems, building upon previous works. We go on to categorise existing benchmarks for ZSG, as well as current methods for tackling these problems. Finally, we provide a critical discussion of the current state of the field, including recommendations for future work. Among other conclusions, we argue that taking a purely procedural content generation approach to benchmark design is not conducive to progress in ZSG, we suggest fast online adaptation and tackling RL-specific problems as some areas for future work on methods for ZSG, and we recommend building benchmarks in underexplored problem settings such as offline RL ZSG and reward-function variation.
BibTeX
@article{kirk2023survey,
  author = {Robert Kirk and Amy Zhang and Edward Grefenstette and Tim Rocktäschel},
  title = {{A Survey of Generalisation in Deep Reinforcement Learning}},
  journal = {J. Artif. Intell. Res.},
  year = {2023},
  url = {https://openreview.net/forum?id=yINUEWrD2Rv},
}

Loading related content...

Figure