The Paradox of Choice: On the Role of Attention in Hierarchical Reinforcement Learning

Andrei Cristian Nica, Khimya Khetarpal, Doina PrecupAttention @ NeurIPS 2022 · OpenReview

TL;DR

This paper introduces "affordances" as a form of "hard attention" in hierarchical reinforcement learning (HRL), which restricts an agent's choices to only those that are feasible and likely to lead to a subgoal. They demonstrate that this focused attention, despite offering fewer options, can significantly improve learning speed and performance, highlighting the "paradox of choice" in AI decision-making.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents often struggle with two fundamental challenges: long planning horizons (making decisions that have consequences far into the future) and a high branching factor (having too many choices at each decision point). Hierarchical Reinforcement Learning (HRL) tackles the first by allowing agents to learn temporally extended actions (options) that skip over multiple steps. However, HRL can exacerbate the second problem, as the number of available options can multiply. This paper addresses how to effectively manage this "breadth" of choices in HRL. The authors propose that by restricting an agent's attention to a smaller, more relevant set of options, akin to how humans focus on what's important, learning can become more efficient and effective.

Methods

The core contribution is the characterization of "affordances" as a "hard attention" mechanism for action choices in HRL. This contrasts with "soft attention," where an agent assigns preferences to all options but doesn't eliminate any.

  • Affordances as Hard Attention: The paper defines affordances based on an "intent completion function" that estimates the probability of reaching a specified subgoal within a given option. A hard threshold on this function determines which options are "affordable" (i.e., considered by the agent). This effectively prunes the action space.
  • Affordance-Aware Subgoal Options: They propose an online, model-free algorithm to learn these affordances and associated subgoal options simultaneously. The algorithm learns an "intent completion network" (ICnet) that predicts the discounted intent completion for state-option pairs. This ICnet's predictions are then used to derive affordances (hard attention) or interest functions (soft attention).
  • Learning Options and Attention: The algorithm trains option policies (using PPO) and termination conditions, as well as the ICnet, concurrently. This allows the attention mechanism to adapt as options are learned.

Datasets

The paper uses two main environments for evaluation:

  • MiniGrid: A discrete gridworld environment used for experiments on data generation, abstract value learning, and handling increasing numbers of choices.
  • Fetch Domain: A continuous control robotics environment (FetchReach-v0) adapted for tasks with multiple goals, used to demonstrate the scalability of their approach.

No new benchmarks were introduced; existing environments were adapted to suit the experimental needs.

Key Insights and Ideas

  • The "Paradox of Choice" in RL: The central finding is that limiting an RL agent's choices through hard attention (affordances) can lead to better performance and faster learning, contrary to the intuition that more options are always better.
  • Affordances as a Powerful Pruning Mechanism: By focusing on options that are genuinely likely to achieve a subgoal, agents can avoid exploring less promising paths, leading to more efficient learning.
  • Hard Attention for Credit Assignment: In long-horizon, sparse reward tasks, hard attention seems to improve credit assignment by helping the agent focus on relevant sequences of actions, leading to better performance than soft attention or no attention.
  • Synergy between Attention and HRL: The paper demonstrates that learning attention mechanisms concurrently with HRL options is crucial for effective performance, allowing the agent to adapt its focus as it learns.

Key Results

  • Improved Sample Efficiency in Data Generation: Hard attention led to faster convergence of learned option policies during data generation compared to soft attention or no attention (Figure 2).
  • Superior Performance in Long-Horizon Tasks: In challenging long-horizon sparse reward tasks with increasing numbers of objects (and thus complexity), HRL with hard attention consistently outperformed HRL with soft attention, no attention, and even a flat agent with pseudo-rewards (Figure 3). In the hardest task, only the hard-attention agent was able to learn.
  • Benefit with Increasing Choices: As the number of available options increased (from 6 to 12), the performance gap between hard and soft attention widened, with hard attention showing better results, further supporting the paradox of choice (Figure 4).
  • Scalability to Continuous Control: The approach showed an advantage in learning in a continuous control robotics domain (FetchReach-v0) with multiple goals, demonstrating its applicability beyond discrete environments (Figure 6).

Limitations and Future Work

  • Subgoal Assumption: The current work assumes that subgoals are provided. The authors acknowledge that learning intents, affordances, and options simultaneously would be more challenging.
  • Future Work - Subgoal Discovery: A natural extension is to relax the subgoal assumption by incorporating subgoal discovery mechanisms, potentially through an iterative approach combining subgoal discovery and affordance-aware option learning.
  • Future Work - Soft Attention Variants: While hard attention showed strong results, the authors mention that exploring variations of soft attention, such as softmax with temperature, could facilitate smoother transitions and potentially offer benefits in other scenarios.
  • Future Work - Domain Knowledge: The effectiveness of the approach relies on having good subgoals, which are currently derived from domain knowledge. Future work could explore how to generalize this to scenarios with less explicit domain knowledge.

Summary updated

Abstract

Decision-making AI agents are often faced with two important challenges: the depth of the planning horizon, and the branching factor due to having many choices. Hierarchical reinforcement learning methods aim to solve the first problem, by providing shortcuts that skip over multiple time steps. To cope with the breadth, it is desirable to restrict the agent's attention at each step to a reasonable number of possible choices. The concept of affordances (Gibson, 1977) suggests that only certain actions are feasible in certain states. In this work, we first characterize "affordances" as a "hard" attention mechanism that strictly limits the available choices of temporally extended options. We then investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices. To this end, we present an online, model-free algorithm to learn affordances that can be used to further learn subgoal options. Finally, we identify and empirically demonstrate the settings in which the "paradox of choice" arises, i.e. when having fewer but more meaningful choices improves the learning speed and performance of a reinforcement learning agent.
BibTeX
@inproceedings{nica2022paradox,
  author = {Andrei Cristian Nica and Khimya Khetarpal and Doina Precup},
  title = {{The Paradox of Choice: On the Role of Attention in Hierarchical Reinforcement Learning}},
  booktitle = {Attention @ NeurIPS},
  year = {2022},
  url = {https://openreview.net/forum?id=WfsYwoltD2},
}
Figure