TL;DR
This paper introduces switching successor measures, a mathematical framework showing that hierarchical control structure is already implicitly encoded within standard successor representations. Leveraging this insight, the authors propose FB $\pi$-Switch, an offline reinforcement learning algorithm that extracts both high-level subgoal planning and low-level control from a single learned representation, enabling zero-shot generalization to arbitrary, general reward functions without requiring fixed time horizons or manual subgoal design.
What Problem Does This Paper Address?
Decomposing long-horizon tasks into smaller, manageable subproblems is the core promise of Hierarchical Reinforcement Learning (HRL). However, most current HRL methods suffer from two major limitations:
- Goal-Conditioned Restrictions: They are largely confined to goal-reaching tasks (e.g., "go to state X") and struggle with general RL problems involving complex or distributed reward functions (e.g., "collect coins while avoiding traps").
- Rigid Temporal Abstractions: State-of-the-art methods like HIQL rely on a fixed temporal horizon (e.g., picking a subgoal exactly $k$ steps away). In complex or uneven environments, a fixed step horizon introduces severe estimation bias because the dataset might lack optimal trajectories that perfectly match that exact window.
The authors tackle a fundamental question: Can we build a unified foundation where both zero-shot reward generalization and hierarchical control naturally emerge from the same learned representation?
Methods
The authors introduce switching successor measures to replace fixed-horizon subgoals with hitting-time switching—meaning the agent follows a subgoal policy until it actually reaches the subgoal, then switches to the global policy.
Remarkably, Theorem 1 proves that the switching successor measures required for high-level planning can be derived mathematically from a single, standard classical successor measure. No additional representation learning is needed to build the hierarchy!
Based on this, they develop FB $\pi$-Switch, which trains in three distinct stages:
- Learning State-Successor Representations: It learns action-free, reward-conditioned representations using an expectile regression objective. This allows single-step learning without explicit policy feedback or coupled optimization loops.
- Learning High-Level Policy ($\pi^h$): The high-level policy learns to select latent subgoals ($w$) by maximizing a proxy of the switching advantage function derived from Stage 1.
- Learning Low-Level Policy ($\pi^\ell$): The low-level policy learns to execute primitive actions to reach those subgoals using Advantage-Weighted Regression (AWR).
Datasets
The method was evaluated on:
- Discrete Maze Environment: A custom 104-state gridworld used for exact mathematical validation and qualitative analysis of the learned advantage functions.
- OGBench AntMaze: A challenging continuous control benchmark featuring a 29-dimensional state space and 8-dimensional action space. The authors utilized the
navigate-v0offline datasets across four maze scales: Medium, Large, Giant, and Teleport.
Key Insights and Ideas
- Hierarchy is Already There: The most profound insight is that standard successor representations implicitly contain hierarchical structure. You don't need to train separate high-level and low-level representations; you just need the right mathematical tool to extract them.
- The Danger of Fixed Horizons: Fixed-horizon subgoal selection ($k$-steps) creates a mismatch in anisotropic environments (where moving in some directions is harder than others) or when data is sub-optimal. Hitting-time switching bypasses this entirely.
- Coherent Trajectories: Unlike goal-conditioned baselines that pick subgoals based on immediate local actions, FB $\pi$-Switch uses goal-only representations. This forces the high-level policy to generate subgoals that form smooth, coherent paths toward the final objective.
Key Results
Goal-Conditioned Tasks
In standard goal-reaching AntMaze tasks, FB $\pi$-Switch performs comparably to HIQL (which is explicitly tailored for goal-reaching) and significantly outperforms non-hierarchical baselines. For instance, in the Large AntMaze, FB $\pi$-Switch achieves a 66% success rate, compared to just 21% for standard Forward-Backward (FB) representations.
General Reward Tasks
When evaluated on complex, distributed reward landscapes (incorporating positive rewards, high-value goals, and negative penalty regions), FB $\pi$-Switch demonstrates superior robustness.
As shown by the Interquartile Mean (IQM) scores, FB $\pi$-Switch consistently achieves the highest average performance across tasks, outperforming both ICVF and One-Step FB baselines. Deeper analysis revealed that the hierarchical variant succeeds primarily because it navigates to the highest-rewarding states significantly faster than its non-hierarchical counterpart.
Limitations and Future Work
- Advantage Function Approximation: In practice, accurately estimating the local component of the exact switching advantage function is highly challenging because it requires isolating states with non-zero rewards. The authors had to use a simplified proxy (dropping a term) to make it stable.
- Representation Dependency: The performance of the hierarchy remains tightly bound to the quality of the underlying base representation model. If the initial successor measure learning fails to capture the environment's dynamics, the extracted hierarchy degrades.
- Multi-Subgoal Planning: Currently, the agent plans one subgoal ahead. Extending this framework to dynamically chain sequences of multiple subgoals remains an open challenge due to compounding errors.
Summary updated