Automated Curriculum Learning by Rewarding Temporally Rare Events

Niels Justesen, Sebastian Risi

This paper, "Automated Curriculum Learning by Rewarding Temporally Rare Events," presents a clever shift in how we think about training Reinforcement Learning (RL) agents. Instead of telling an agent exactly what the goal is, the authors suggest we should simply reward it for doing things it hasn't done much lately.

TL;DR

The authors introduce Rarity of Events (RoE), a method that automates curriculum learning by rewarding agents for experiencing "temporally rare" events. By ignoring extrinsic rewards and focusing on rarity, agents learn more versatile, adaptable behaviors that outperform standard algorithms in complex, sparse-reward environments like ViZDoom.

What Problem Does This Paper Address?

Reinforcement Learning often fails when rewards are "sparse"—meaning the agent wanders around for a long time without receiving any feedback. To fix this, researchers usually rely on reward shaping (manually adding "breadbox" rewards) or curriculum learning (starting with easy tasks and getting harder).

The problem? Both require massive amounts of human effort and domain expertise to design. If you change the environment slightly, your manual curriculum might break. The authors wanted a "set it and forget it" way to guide an agent through a complex environment without needing a human to hand-craft the reward schedule.

Methods

The core innovation is the Rarity of Events (RoE) reward function.

  1. Event Tracking: The researchers define a list of 26 "events" (e.g., picking up a medkit, firing a weapon, moving one unit of distance, killing an enemy).
  2. The Rarity Formula: The reward for an event is inversely proportional to how often it has occurred in the recent past (tracked via a sliding window buffer of the last 100 episodes).
  3. If an event is common (like moving), its reward drops toward zero.
  4. If an event is rare (like killing a specific enemy), its reward is high.
  5. Intrinsic Motivation: Crucially, during training, the agent completely ignores the game's actual score (extrinsic reward). It is motivated purely by "curiosity" for rare events.
  6. Algorithm: They integrated RoE with Advantage Actor-Critic (A2C), a standard deep RL algorithm, using raw pixels as input.

This creates an automated curriculum: early in training, the agent gets rewards just for moving. Once it "gets bored" of moving (because it's no longer rare), it seeks out new events, like picking up items or shooting, naturally progressing to more difficult tasks.

Datasets

The method was tested using the ViZDoom framework, an RL research platform based on the classic game Doom. They used five increasingly difficult scenarios:

  • Health Gathering (and Supreme): Survive on acid floors by finding medkits in a room or maze.
  • My Way Home: Find armor in a maze.
  • Deadly Corridor: Reach armor at the end of a hallway guarded by enemies.
  • Deathmatch: A large arena with multiple weapon types, ammo, and enemies.

To test adaptability, they also created five variations of the Deathmatch map where only one specific weapon type was available (e.g., only the Chainsaw or only the Rocket Launcher).

Key Insights and Ideas

  • The "Boredom" Driver: RoE prevents agents from getting stuck in "local optima." In standard RL, an agent might find one safe corner and stay there. In RoE, staying in a corner becomes "common" and unrewarding, forcing the agent to go out and find something new.
  • Versatility vs. Optimization: Standard A2C often becomes a "specialist." In Deathmatch, it learned to camp the most powerful weapon (the Plasma Gun) and ignore everything else. RoE agents became "jacks-of-all-trades," learning to use every weapon because using only one would make its reward drop.
  • Adaptability is a Side Effect of Rarity: Because RoE agents are forced to experience a wide variety of events to keep their rewards high, they are much better at handling changes. If the "best" weapon is removed, the RoE agent already knows how to use the others; the standard A2C agent often just wanders aimlessly.

Key Results

  • Outperformance: RoE outperformed the baseline A2C in 4 out of 5 scenarios. In "Health Gathering Supreme" (the maze), RoE achieved a score of 1427, while standard A2C struggled at 305.
  • Zero-Shot Adaptation: In the Deathmatch variations (where the "best" weapon was removed), RoE was significantly more robust. For example, in the "Chainsaw only" map, RoE scored 3750 while the baseline A2C dropped to 1025.
  • Statistical Significance: The improvements were highly significant ($p < 0.0001$) in almost all cases.

Limitations and Future Work

  • Event Definition: While RoE automates the curriculum, a human still has to define what constitutes an "event." The authors suggest future work could involve learning these event definitions automatically from raw data.
  • Negative Events: The authors note that you shouldn't reward every event. For example, rewarding "wasting ammo" might lead to poor play. Choosing the right "positive" events is still a bit of an art.
  • Scaling: The study used 26 events. It remains to be seen how RoE scales to games like StarCraft II, which might have hundreds or thousands of possible event types.
  • Combining Rewards: A promising next step is combining RoE (for exploration) with the actual game reward (for optimization) to get the best of both worlds.

Summary updated

Abstract

Reward shaping allows reinforcement learning (RL) agents to accelerate learning by receiving additional reward signals. However, these signals can be difficult to design manually, especially for complex RL tasks. We propose a simple and general approach that determines the reward of pre-defined events by their rarity alone. Here events become less rewarding as they are experienced more often, which encourages the agent to continually explore new types of events as it learns. The adaptiveness of this reward function results in a form of automated curriculum learning that does not have to be specified by the experimenter. We demonstrate that this Rarity of Events (RoE) approach enables the agent to succeed in challenging VizDoom scenarios without access to the extrinsic reward from the environment. Furthermore, the results demonstrate that RoE learns a more versatile policy that adapts well to critical changes in the environment. Rewarding events based on their rarity could help in many unsolved RL environments that are characterized by sparse extrinsic rewards but a plethora of known event types.
BibTeX
@inproceedings{justesen2018automated,
  author = {Niels Justesen and Sebastian Risi},
  title = {{Automated Curriculum Learning by Rewarding Temporally Rare Events}},
  booktitle = {CIG},
  year = {2018},
  url = {https://openreview.net/forum?id=ATw9-aU1Y1V},
}

Loading related content...

Figure