Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning

Stefan Stojanovic, Alexandre ProutierearXiv 2026 · arXiv

TL;DR

This paper introduces switching successor measures, a mathematical framework showing that hierarchical control structure is already implicitly encoded within standard successor representations. Leveraging this insight, the authors propose FB $\pi$-Switch, an offline reinforcement learning algorithm that extracts both high-level subgoal planning and low-level control from a single learned representation, enabling zero-shot generalization to arbitrary, general reward functions without requiring fixed time horizons or manual subgoal design.

What Problem Does This Paper Address?

Decomposing long-horizon tasks into smaller, manageable subproblems is the core promise of Hierarchical Reinforcement Learning (HRL). However, most current HRL methods suffer from two major limitations:

  1. Goal-Conditioned Restrictions: They are largely confined to goal-reaching tasks (e.g., "go to state X") and struggle with general RL problems involving complex or distributed reward functions (e.g., "collect coins while avoiding traps").
  2. Rigid Temporal Abstractions: State-of-the-art methods like HIQL rely on a fixed temporal horizon (e.g., picking a subgoal exactly $k$ steps away). In complex or uneven environments, a fixed step horizon introduces severe estimation bias because the dataset might lack optimal trajectories that perfectly match that exact window.

The authors tackle a fundamental question: Can we build a unified foundation where both zero-shot reward generalization and hierarchical control naturally emerge from the same learned representation?

Methods

The authors introduce switching successor measures to replace fixed-horizon subgoals with hitting-time switching—meaning the agent follows a subgoal policy until it actually reaches the subgoal, then switches to the global policy.

Remarkably, Theorem 1 proves that the switching successor measures required for high-level planning can be derived mathematically from a single, standard classical successor measure. No additional representation learning is needed to build the hierarchy!

Based on this, they develop FB $\pi$-Switch, which trains in three distinct stages:

  1. Learning State-Successor Representations: It learns action-free, reward-conditioned representations using an expectile regression objective. This allows single-step learning without explicit policy feedback or coupled optimization loops.
  2. Learning High-Level Policy ($\pi^h$): The high-level policy learns to select latent subgoals ($w$) by maximizing a proxy of the switching advantage function derived from Stage 1.
  3. Learning Low-Level Policy ($\pi^\ell$): The low-level policy learns to execute primitive actions to reach those subgoals using Advantage-Weighted Regression (AWR).
FB $\pi$-Switch training pipeline

Datasets

The method was evaluated on:

  • Discrete Maze Environment: A custom 104-state gridworld used for exact mathematical validation and qualitative analysis of the learned advantage functions.
  • OGBench AntMaze: A challenging continuous control benchmark featuring a 29-dimensional state space and 8-dimensional action space. The authors utilized the navigate-v0 offline datasets across four maze scales: Medium, Large, Giant, and Teleport.

Key Insights and Ideas

  • Hierarchy is Already There: The most profound insight is that standard successor representations implicitly contain hierarchical structure. You don't need to train separate high-level and low-level representations; you just need the right mathematical tool to extract them.
  • The Danger of Fixed Horizons: Fixed-horizon subgoal selection ($k$-steps) creates a mismatch in anisotropic environments (where moving in some directions is harder than others) or when data is sub-optimal. Hitting-time switching bypasses this entirely.
Fixed horizon bias vs optimal path selection
  • Coherent Trajectories: Unlike goal-conditioned baselines that pick subgoals based on immediate local actions, FB $\pi$-Switch uses goal-only representations. This forces the high-level policy to generate subgoals that form smooth, coherent paths toward the final objective.
Subgoal visualization comparison

Key Results

Goal-Conditioned Tasks

In standard goal-reaching AntMaze tasks, FB $\pi$-Switch performs comparably to HIQL (which is explicitly tailored for goal-reaching) and significantly outperforms non-hierarchical baselines. For instance, in the Large AntMaze, FB $\pi$-Switch achieves a 66% success rate, compared to just 21% for standard Forward-Backward (FB) representations.

General Reward Tasks

When evaluated on complex, distributed reward landscapes (incorporating positive rewards, high-value goals, and negative penalty regions), FB $\pi$-Switch demonstrates superior robustness.

Normalized returns across general reward tasks

As shown by the Interquartile Mean (IQM) scores, FB $\pi$-Switch consistently achieves the highest average performance across tasks, outperforming both ICVF and One-Step FB baselines. Deeper analysis revealed that the hierarchical variant succeeds primarily because it navigates to the highest-rewarding states significantly faster than its non-hierarchical counterpart.

Limitations and Future Work

  • Advantage Function Approximation: In practice, accurately estimating the local component of the exact switching advantage function is highly challenging because it requires isolating states with non-zero rewards. The authors had to use a simplified proxy (dropping a term) to make it stable.
  • Representation Dependency: The performance of the hierarchy remains tightly bound to the quality of the underlying base representation model. If the initial successor measure learning fails to capture the environment's dynamics, the extracted hierarchy degrades.
  • Multi-Subgoal Planning: Currently, the agent plans one subgoal ahead. Extending this framework to dynamically chain sequences of multiple subgoals remains an open challenge due to compounding errors.

Summary updated

Abstract

Hierarchical reinforcement learning can improve generalization by decomposing long-horizon decision-making into simpler subproblems. However, existing approaches often rely on restrictive design choices, such as fixed temporal abstractions or goal-conditioned objectives, which largely confine them to goal-reaching tasks and limit their applicability to general reward functions. In this paper, we introduce switching successor measures, an extension of successor measures that enables hierarchical control in zero-shot reinforcement learning without additional supervision, fixed horizons, or manually designed subgoals. We show that switching successor measures arise naturally from classical successor measures while preserving their underlying structure. Building on this result, we propose FB $π$-Switch, an algorithm that extracts both a high-level subgoal-selection policy and a low-level control policy directly from forward-backward (FB) representations, allowing hierarchical behavior to emerge from a single learned representation. Experiments on both goal-conditioned and general reward-based tasks show that FB $π$-Switch improves over non-hierarchical baselines and matches state-of-the-art hierarchical methods in goal-conditioned settings. These results demonstrate that structured successor representations provide a flexible foundation for hierarchical zero-shot reinforcement learning beyond goal-reaching tasks. Our project website is available at: https://stestokth.github.io/switching-successors/.
BibTeX
@misc{stojanovic2026switching,
  author = {Stefan Stojanovic and Alexandre Proutiere},
  title = {{Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning}},
  year = {2026},
  eprint = {2605.13207},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.13207},
}
Figure