Representation-driven Option Discovery in Reinforcement Learning

Marlos C. Machado

Abstract

Reinforcement Learning (RL) agents often struggle with long-horizon tasks requiring complex sequences of actions. This paper presents the Representation-driven Option Discovery (ROD) cycle, a framework that integrates representation learning and temporal abstraction. By using internal world models to discover "options" (macro-actions), agents can explore more effectively, generating richer data that further refines their representations. This iterative process creates a virtuous cycle, enabling agents to autonomously build complex behaviors and solve challenging navigation and control tasks.

Introduction

A central challenge in Reinforcement Learning is the "chicken-and-egg" problem of exploration and abstraction. To solve long-horizon tasks, agents benefit from the options framework, which allows for macro-actions like "exit the room" rather than low-level motor commands. However, most research assumes these options are pre-defined. Discovering useful options autonomously requires a good representation of the environment, yet a good representation often requires the broad exploration that options provide.

The Representation-driven Option Discovery (ROD) framework addresses this by linking representation learning and option discovery into a single, self-improving loop. Instead of wandering aimlessly, the agent uses its evolving understanding of environmental geometry to define and solve intrinsic subtasks.

The ROD Cycle

The core of the proposed framework is the ROD Cycle, an iterative process consisting of four primary stages that allow an agent to construct increasingly complex temporal abstractions.

The Representation-driven Option Discovery (ROD) cycle.
  1. Interaction: The agent interacts with the environment to collect experience data.
  2. Representation Learning: This data updates the agent’s internal model. A key component is the Successor Representation (SR), which captures environmental geometry by predicting future state occupancy.
  3. Subtask Definition: The agent analyzes the learned representation to identify intrinsic goals. By examining the eigenvectors of the representation (inspired by "eigenoptions"), the agent can identify structural bottlenecks, such as doorways or hallways, which serve as natural subtask targets.
  4. Option Discovery: The agent learns policies to achieve these intrinsic goals. These policies are added to the agent's toolkit as new "options," empowering it to explore more effectively in the next iteration.

To scale this approach to high-dimensional state spaces, the framework utilizes neural networks to estimate the eigenfunctions of the graph Laplacian, moving beyond simple grid-world environments to complex 3D tasks.

Key Insights and Results

The ROD framework is rooted in a "constructivist" philosophy of AI, where agents build hierarchical abstractions over time. By leveraging the Successor Representation and Laplacian eigenfunctions, the agent discovers the "topology" of its environment purely through movement statistics, without requiring external reward signals.

Research conducted between 2017 and 2024 demonstrates several key advantages:

  • Efficient Exploration: Eigenoptions significantly reduce the time needed to cover an environment compared to random-walk exploration.
  • Benchmark Performance: Deep Laplacian-based options have achieved state-of-the-art (SOTA) results on standard benchmarks, including Atari 2600 games and complex 3D navigation tasks.
  • Reward Maximization: By solving the fundamental exploration problem, agents become significantly more efficient at learning policies that maximize task-specific rewards.

Limitations and Future Directions

While the ROD cycle offers a robust framework for autonomous discovery, several challenges remain. Scaling to highly non-linear and dynamic environments using neural networks requires careful engineering to maintain stability. Additionally, there is a computational trade-off between the time spent "discovering" options and the time spent "exploiting" them for task rewards. Future work aims to automate the determination of optimal abstraction levels and the number of options needed for a given environment.

Summary updated

Abstract

The ability to reason at multiple levels of temporal abstraction is a fundamental aspect of intelligence. In reinforcement learning (RL), this attribute is often modelled through temporally extended courses of actions called options. In this talk, I will introduce a general framework for option discovery, which uses the agent's representation to discover useful options. By leveraging these options to generate a rich stream of experience, the agent can improve its representations and learn more effectively. This representation-driven option discovery approach creates a virtuous cycle of refinement, continuously improving both the representation and options, and it is particularly effective for problems where agents need to operate at varying levels of abstraction to succeed.
BibTeX
@inproceedings{machado2025representation,
  author = {Marlos C. Machado},
  title = {{Representation-driven Option Discovery in Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i27.35100},
}
Figure