Classifying Options for Deep Reinforcement Learning

Kai Arulkumaran, Nat Dilokthanakul, Murray Shanahan, Anil Anthony Bharath

In the quest for more efficient artificial intelligence, researchers often look to how humans solve complex problems: by breaking them down into smaller, manageable subtasks. This paper, "Classifying Options for Deep Reinforcement Learning," explores how to bake this hierarchical intuition directly into the architecture of Deep Q-Networks (DQNs).

TL;DR

The authors combine the "options framework" for hierarchical reinforcement learning with Deep Q-Networks by using multiple "option heads" and a supervisory network. This architectural shift significantly reduces the number of samples needed to learn conflicting subtasks (negative transfer) without sacrificing performance on complementary ones.

What Problem Does This Paper Address?

Standard Deep Reinforcement Learning (RL) agents, like the classic DQN, are notoriously "sample hungry"—they require millions of interactions with an environment to learn effective policies. While a standard DQN might implicitly learn to handle different parts of a task, it often struggles with negative transfer. This occurs when learning one subtask (e.g., "collect the power-up") interferes with the representation needed for another (e.g., "avoid the ghost").

The authors investigate whether explicitly structuring a network to handle distinct subtasks—imposing "structural priors"—can make learning more efficient and robust against such interference.

Methods

The researchers propose an augmented DQN architecture that splits the policy into two main components:

  1. Option Heads: Instead of a single output layer, the network features multiple "heads" branching off from shared convolutional layers. Each head represents a specific "option" or sub-policy. By limiting the number of neurons in these heads, the authors force the network to specialize.
  2. Supervisory Network: This is a classifier that observes the state and decides which option head should be in control.
Figure 1: Comparison of DQN architectures. a) Standard DQN. b) DQN with 2 option heads.

During training, an "oracle" (ground truth knowledge of the subtask) tells the agent which head to use and which experience replay buffer to update. During evaluation, the supervisory network takes over the oracle's job. This setup is compared against a standard DQN and a "Half DQN" (which has the same number of parameters as a single option head) to ensure the benefits come from the structure, not just a change in parameter count.

Datasets

The authors evaluated their approach using a modified version of the game "Catch." In this environment, a paddle at the bottom of a 24x24 grid must move to catch falling pixels.

To test transfer, they introduced two scenarios:

  • Positive Transfer: The agent must catch both white and grey balls (both give +1 reward). The subtasks are complementary.
  • Negative Transfer: The agent must catch grey balls (+1 reward) but avoid white balls (-1 reward). Here, the subtasks are in direct conflict.
Figure 2: 8 frames of Catch. The first row shows the white ball subtask, and the second row shows the grey ball subtask.

Key Insights and Ideas

  • Structural Priors as Regularizers: Explicitly separating the policy into heads acts as a form of regularization. It prevents the "gradient interference" that happens when a single network tries to map similar states to wildly different actions.
  • Shared vs. Specific Knowledge: The shared convolutional layers learn general features (like "where is the ball?" and "how do I move the paddle?"), while the heads learn the specific logic of the subtask (like "move toward the ball" vs. "move away").
  • Capacity vs. Architecture: The experiments revealed that for simple, non-conflicting tasks, simply increasing the number of neurons (capacity) is the most important factor. However, when tasks conflict, the architecture (how those neurons are organized) becomes the deciding factor for efficiency.

Key Results

The most striking results appeared in the negative transfer experiments.

  • Lower Sample Complexity: The DQN with option heads reached peak performance significantly faster than the standard DQN in conflicting scenarios.
  • Robustness to Interference: While the standard DQN eventually learned the negative transfer task given enough capacity, the option heads architecture demonstrated a much smoother and faster learning curve.
  • No Penalty for Simplicity: In the positive transfer (complementary) tasks, the option heads architecture performed just as well as the standard baseline, proving that the added structure doesn't hinder the agent when tasks are simple.
Figure 3: The average score per episode of the standard DQN, half DQN and DQN with 2 option heads. The average and standard deviation are calculated over 15 runs (with different random seeds), with the score for each run being averaged over 250 episodes. Each epoch corresponds to 10000 training steps. The number of neurons corresponds to the number of neurons in the fully connected hidden layer of the standard DQN. In the positive transfer setting the most important factor is capacity, not the architecture. As capacity increases the difference in performance between the networks diminishes. In the negative transfer setting the effect of capacity is strong when capacity is very low, but otherwise the DQN with option heads demonstrates a superior sample complexity over the baselines. Best viewed in colour.

Limitations and Future Work

  • Reliance on an Oracle: The current model requires an "oracle" during training to define when one subtask ends and another begins. In many real-world applications, these boundaries are fuzzy or unknown.
  • Option Discovery: The authors note that the next logical step is option discovery—enabling the agent to learn how to partition the state space and create its own subtasks and supervisory signals without human intervention.
  • Task Complexity: While "Catch" is a great proof-of-concept, further research is needed to see how this architecture scales to more complex environments with many more overlapping or hierarchical subtasks.

Summary updated

Abstract

In this paper we combine one method for hierarchical reinforcement learning - the options framework - with deep Q-networks (DQNs) through the use of different "option heads" on the policy network, and a supervisory network for choosing between the different options. We utilise our setup to investigate the effects of architectural constraints in subtasks with positive and negative transfer, across a range of network capacities. We empirically show that our augmented DQN has lower sample complexity when simultaneously learning subtasks with negative transfer, without degrading performance when learning subtasks with positive transfer.
BibTeX
@misc{arulkumaran2016classifying,
  author = {Kai Arulkumaran and Nat Dilokthanakul and Murray Shanahan and Anil Anthony Bharath},
  title = {{Classifying Options for Deep Reinforcement Learning}},
  year = {2016},
  eprint = {1604.08153},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1604.08153},
}
Figure