Automated Curriculum Learning by Rewarding Temporally Rare Events

Niels Justesen, Sebastian Risi

Abstract: Reward shaping allows reinforcement learning (RL) agents to accelerate learning by receiving additional reward signals. However, these signals can be difficult to design manually, especially for complex RL tasks. We propose a simple and general approach that determines the reward of pre-defined events by their rarity alone. Here events become less rewarding as they are experienced more often, which encourages the agent to continually explore new types of events as it learns. The adaptiveness of this reward function results in a form of automated curriculum learning that does not have to be specified by the experimenter. We demonstrate that this Rarity of Events (RoE) approach enables the agent to succeed in challenging VizDoom scenarios without access to the extrinsic reward from the environment. Furthermore, the results demonstrate that RoE learns a more versatile policy that adapts well to critical changes in the environment. Rewarding events based on their rarity could help in many unsolved RL environments that are characterized by sparse extrinsic rewards but a plethora of known event types.

Introduction

Reinforcement Learning (RL) often fails in environments where rewards are "sparse," meaning an agent may explore for extended periods without receiving any feedback. Traditionally, researchers have addressed this through manual reward shaping or human-designed curriculum learning. However, these methods require significant domain expertise and are often fragile to environmental changes. This paper introduces Rarity of Events (RoE), a method that automates curriculum learning by rewarding agents for experiencing "temporally rare" events, effectively driving exploration through an intrinsic sense of "boredom."

The Rarity of Events (RoE) Methodology

The RoE approach shifts the focus from optimizing an extrinsic score to seeking novelty. The framework operates on three core principles:

  1. Event Tracking: A list of 26 discrete events is defined (e.g., picking up a medkit, firing a weapon, or moving a specific distance).
  2. The Rarity Formula: The reward for any given event is inversely proportional to its frequency in the recent past, tracked via a sliding window buffer of the last 100 episodes. Common events (like moving) eventually yield zero reward, while rare events (like defeating a specific enemy) provide high rewards.
  3. Intrinsic Motivation: During training, the agent completely ignores the environment's extrinsic reward (the game score). It is motivated solely by the curiosity generated by the RoE function.

This creates a natural, automated curriculum. Early in training, the agent is rewarded for basic movement. As those actions become common and their rewards diminish, the agent is forced to seek out more complex interactions, such as item collection or combat, to maintain its reward intake.

Experimental Setup

The methodology was evaluated using the ViZDoom framework across five scenarios of increasing complexity, ranging from simple survival on acid floors to complex multi-weapon deathmatches.

Map layouts for the five ViZDoom scenarios used to evaluate RoE, including Deathmatch and Health Gathering Supreme.

Key Results and Adaptability

RoE demonstrated superior performance and robustness compared to standard Advantage Actor-Critic (A2C) algorithms. In the "Health Gathering Supreme" maze, RoE achieved a score of 1427, vastly outperforming the standard A2C score of 305.

Performance comparison in the Deadly Corridor scenario; standard A2C fails to learn (blue line at zero), while A2C+RoE (red line) successfully navigates the sparse reward environment.

Beyond raw scores, RoE agents proved to be more versatile. While standard agents often became "specialists"—for instance, only learning to use the most powerful weapon in a Deathmatch—RoE agents became "jacks-of-all-trades." Because using a single weapon repeatedly makes that event "common" and less rewarding, RoE agents naturally learned to utilize the entire arsenal. This versatility allowed for successful zero-shot adaptation; when the primary weapon was removed from the environment, RoE agents maintained high performance while standard agents often failed entirely.

Discussion and Future Work

While RoE successfully automates the curriculum, it currently relies on human-defined event types. Future research could explore methods to learn these event definitions automatically from raw pixel data. Additionally, the selection of which events to reward remains a critical design choice, as rewarding counter-productive actions could hinder performance. Nevertheless, RoE represents a significant step toward general-purpose RL agents capable of navigating complex, sparse-reward environments without manual intervention.

Summary updated

Abstract

Reward shaping allows reinforcement learning (RL) agents to accelerate learning by receiving additional reward signals. However, these signals can be difficult to design manually, especially for complex RL tasks. We propose a simple and general approach that determines the reward of pre-defined events by their rarity alone. Here events become less rewarding as they are experienced more often, which encourages the agent to continually explore new types of events as it learns. The adaptiveness of this reward function results in a form of automated curriculum learning that does not have to be specified by the experimenter. We demonstrate that this \emph{Rarity of Events} (RoE) approach enables the agent to succeed in challenging VizDoom scenarios without access to the extrinsic reward from the environment. Furthermore, the results demonstrate that RoE learns a more versatile policy that adapts well to critical changes in the environment. Rewarding events based on their rarity could help in many unsolved RL environments that are characterized by sparse extrinsic rewards but a plethora of known event types.
BibTeX
@inproceedings{justesen2018automated,
  author = {Niels Justesen and Sebastian Risi},
  title = {{Automated Curriculum Learning by Rewarding Temporally Rare Events}},
  booktitle = {CIG},
  year = {2018},
  url = {https://openreview.net/forum?id=ATw9-aU1Y1V},
}
Figure