Introduction
Reinforcement Learning (RL) agents often struggle in environments where rewards are sparse and layouts are procedurally generated. While Self-Imitation Learning (self-IL) allows agents to learn from their own past successes, it frequently suffers from "behavioral overfitting" in non-static environments. This paper, presented at the 2023 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), introduces novel sampling, filtering, and diversity strategies to overcome these limitations and enhance generalization in complex, sparse-reward tasks.
The Challenge: Sparse Rewards and Procedural Environments
In RL, "sparse rewards" occur when an agent receives no feedback until a task is fully completed, such as navigating a massive maze. Self-Imitation Learning (self-IL) addresses this by storing successful past trajectories in a replay buffer and encouraging the agent to "imitate" its own best moments.
However, most self-IL methods are designed for "singleton" environments where the layout never changes. When applied to Procedurally-Generated (PCG) environments—where every episode features a different layout—standard self-IL often fails. The agent tends to overfit to a specific successful path it found once, failing to learn the general logic required to solve new variations. Previous attempts like RAPID improved this by ranking episodes, but they still replayed data uniformly, treating every step of a successful run as equally important. This leads to behavioral overfitting, where the agent memorizes specific actions rather than learning generalizable skills.
Proposed Methodology
The authors propose three categories of improvements to the self-IL pipeline to foster better generalization:
1. Prioritization Strategies
Instead of uniform sampling from the replay buffer, the researchers tested three proxies to identify the most valuable transitions:
- TD-error: Replaying "surprising" transitions where the agent's value prediction was most inaccurate.
- Log-Likelihood: Promoting frequent, reliable actions to stabilize the policy.
- Novelty: Prioritizing states the agent has rarely visited across the entire training process. This encourages the agent to focus on the mechanics of exploration.
2. Filtering Mechanisms
To prevent the agent from learning from meaningless or redundant data, the authors applied specific filters to the replay buffer:
- Non-zero Return: Only replaying trajectories that actually reached the goal.
- Positive Advantage: Only replaying steps that performed better than the agent’s current average expectation.
- Unique States: Preventing the agent from replaying repetitive loops within a single trajectory, ensuring the imitation is efficient.
3. Diversity Mechanisms
To prevent the agent from getting stuck in local optima, the study utilized two primary diversity techniques:
- Intrinsic Motivation (BeBold): An on-policy novelty-seeking technique that maintains agent curiosity throughout training.
- Forced Diversity: Constraining the buffer to hold a fixed number of episodes from every level encountered. This ensures the agent does not simply "study" the easiest layouts but is exposed to the full breadth of the procedural generation.
Experimental Evaluation
The researchers evaluated their approach on two major PCG benchmarks:
- MiniGrid: Specifically the MultiRoom-N12-S10 (a grueling 12-room navigation task) and ObstructedMaze (which requires manipulating objects like keys and balls).
- ProcGen: The Ninja environment, a high-speed platformer where the agent must navigate elevated platforms and avoid bombs based on raw image inputs.
Key Insights and Results
Novelty as a Catalyst for Learning
Among the prioritization strategies, Novelty consistently outperformed uniform sampling. By focusing on transitions in uncertain state spaces, the agent learns the underlying mechanics of navigation and exploration rather than just memorizing a specific path. In the MultiRoom task, Novelty prioritization combined with Unique state filtering reached the optimal policy with 24% fewer interactions than standard uniform sampling.
The Diversity-Generalization Link
In PCG environments, diversity is a requirement for stability. Without it, prioritization can introduce bias, causing the agent to diverge. Combining self-IL with Intrinsic Motivation (BeBold) acted as a stabilizer, allowing the agent to benefit from prioritization without the risk of overfitting. In the ProcGen Ninja task, "Forced Diversity" made the difference between learning a near-optimal policy and failing to surpass a random agent.
Scaling Imitation
The authors found that increasing the imitation batch size (from 256 to 2048) significantly improved sample efficiency. Larger batches provide a more stable gradient, which is critical when the environment layout is constantly shifting and the agent must find commonalities across diverse experiences.
Conclusion and Future Work
This research establishes a new state-of-the-art performance ceiling for the MiniGrid-MultiRoom-N12-S10 environment. By integrating novelty-based prioritization and forced diversity, the authors demonstrate that self-imitation can be a powerful tool for generalization in procedural environments.
Future work will address the challenges of high-dimensional visual inputs, as current exploration scores are most effective in discrete state spaces. The team also aims to explore how these prioritization techniques pair with other imitation styles, such as Generative Adversarial Imitation Learning (GAIL) or Inverse Reinforcement Learning.
Summary updated