Mitigating Non-Stationarity in Hierarchical Reinforcement Learning

122 concepts92 papers3 proposals

To solve complex, long-horizon tasks, Hierarchical Reinforcement Learning (HRL) breaks a problem down into a multi-level decision-making structure. In a typical two-level hierarchy, a high-level policy (the manager) selects abstract subgoals, while a low-level policy (the worker) executes primitive actions to achieve those subgoals.

While this division of labor is conceptually elegant, training both levels simultaneously introduces a severe mathematical friction known as hierarchical non-stationarity or the moving target problem.

The Anatomy of the Moving Target Problem

In standard reinforcement learning, an agent interacts with an environment governed by a stationary transition probability distribution, $P(s' \mid s, a)$. In HRL, however, the environment seen by the high-level manager is not purely physical; it is mediated by the low-level worker. The high-level transition dynamics can be written as:

$$P_H(s' \mid s, g) = \sum_{a} P(s' \mid s, a) \pi_L(a \mid s, g)$$

Because the low-level policy $\pi_L$ is continuously learning and changing its behavior, the effective transition function $P_H$ shifts under the manager's feet. As noted by Markus Wulfmeier and Dushyant Rao in Data-efficient Hindsight Off-policy Option Learning, this creates an acute challenge for off-policy data reuse. Historical transitions stored in a replay buffer reflect an outdated version of the worker's capabilities, leading to high-variance updates and training instability.

This instability operates as a two-way destabilizing loop:

  1. Low-to-High Instability: Kenneth Marino et al. in Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies observe that when low-level sub-policies adapt concurrently with the high-level controller, the manager receives inconsistent feedback and struggles to associate specific subgoals with consistent outcomes.
  2. High-to-Low Instability: Conversely, as shown by Anders Jonsson et al. in Hierarchical Average-Reward Linearly-solvable Markov Decision Processes, updates to the high-level strategy cause the underlying objectives of low-level subtasks to shift unpredictably, forcing the worker to constantly adapt to changing goals.

According to Gianluca Drappo et al. in Efficient Exploitation of Hierarchical Structure in Sparse Reward Reinforcement Learning, planning against this moving target frequently leads to mathematical instability and unreliable performance.

Subgoal Representation Shift and Latent Collapse

The problem compounds when subgoals are defined within a learned latent space rather than a fixed coordinate system. Siyuan Li et al. demonstrate in Active Hierarchical Exploration with Stable Subgoal Representation Learning that if internal representations shift rapidly, previously learned goal coordinates no longer correspond to the same physical states, causing the high-level policy to fail.

To anchor these representations, Siyuan Li et al. propose in Learning Subgoal Representations with Slow Dynamics to optimize a latent space that prioritizes "slow" features. Forcing the features to evolve at a rate consistent with the manager's decision cycle ensures temporal consistency, giving the worker a stable target space to learn transitions.

Concurrent Training (Unstable):
[Manager Updates] ──(Shifting Subgoals)──> [Worker Faces New Objectives]
       ▲                                              │
       └──────(Shifting Worker Capabilities)──────────┘

Slow-Dynamics / Probabilistic Anchoring (Stable):
[Manager Updates] ──(Slow/Probabilistic Latent Space)──> [Worker Faces Consistent Targets]

Similarly, Vivienne Huiling Wang et al. in Probabilistic Subgoal Representations for Hierarchical Reinforcement Learning mitigate non-stationarity by stabilizing the subgoal space through probabilistic modeling. This ensures high-level actions maintain consistent semantic meanings over time. Without this stabilization, as shown by Haobin Jiang et al. in Discrete Latent Plans via Semantic Skill Abstractions, shifting latent representations can trigger "index collapse," where the model fails to utilize its full repertoire of learned skills and converges to a limited subset of behaviors.

Enforcing Feasibility and Reachability

A direct consequence of non-stationarity is that the manager often issues "impossible" subgoals based on outdated expectations of what the worker can achieve. To prevent this, several frameworks explicitly constrain the manager to the worker's current capabilities.

Joni Pajarinen et al. introduce this constraint in State-Conditioned Adversarial Subgoal Generation via an adversarial discriminator. The discriminator forces the high-level policy to generate subgoals that are strictly reachable by the current low-level agent, dynamically adjusting the goal space as the worker's skills evolve.

Taking a generative approach, Vivienne Huiling Wang in Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals utilizes a conditional diffusion model regularized by a Gaussian Process prior. This allows the manager to quantify its uncertainty regarding goal reachability, preventing the pursuit of impossible objectives and stabilizing the training trajectory.

[Manager Policy] ───(Proposed Subgoal)───> [Reachability Filter / Discriminator]
                                                    │
                                         (Is it achievable NOW?)
                                                    │
                                         ├── Yes ──> [Pass to Worker]
                                         └── No  ──> [Reject / Penalize Manager]

Alternatively, the levels can be decoupled through preference optimization. Souradip Chakraborty et al. in Hierarchical Preference Optimization: Learning to achieve goals via feasible subgoals prediction and Utsav Singh et al. in Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach utilize Direct Preference Optimization (DPO) with a stationary preference dataset for the higher-level policy. By decoupling the manager's learning from the fluctuating reward signals of the evolving worker, they eliminate the moving target problem entirely.

Temporal Filtering and Hindsight Corrections

Another school of thought focuses on synchronizing or filtering the feedback loop between levels.

John Winder et al. in Planning with Abstract Learned Models While Learning Transferable Subtasks propose a synchronization schedule where lower-level subtask models must reach a threshold of certainty before the higher-level planner is allowed to rely on them. This ensures that higher-level policies are built on stable foundations.

In a complementary fashion, Arnaud Robert et al. in Sample Complexity of Goal-Conditioned Hierarchical Reinforcement Learning introduce a filtering mechanism where the high-level policy is updated only when the low-level agent successfully reaches a designated subgoal. This filtering ensures that the high-level agent observes stable transitions, effectively ignoring the noise generated by the low-level agent's initial exploration and failures.

Finally, hindsight mechanisms can retroactively correct for the worker's changing capabilities. As demonstrated by Andrew Levy et al. in Learning Multi-Level Hierarchies with Hindsight, the hierarchy can use hindsight to substitute the subgoals originally attempted with the states the worker actually achieved. This simulates optimal lower-level behavior, grounding subgoals in reality and allowing multiple levels of a hierarchy to learn simultaneously and in parallel without collapsing.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 98 papers
  2. Version 1 · 2026-10-04 · 159 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_non_stationarity_d0ea3f0a,
  title = {hierarchical non-stationarity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-non-stationarity/rdr_d0ea3f0a450441f88cfdd9dbfc678809},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_hierarchical_non_stationarity_d0ea3f0a_v2,
  title = {hierarchical non-stationarity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-non-stationarity/rdr_d0ea3f0a450441f88cfdd9dbfc678809/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake