An Informational Curse of Horizon in Goal-Conditioned Policy Learning

John L. Zhou, Yuxuan Dong, Jonathan C. KaoarXiv 2026 · arXiv

TL;DR

This paper identifies a new "informational curse of horizon" in goal-conditioned reinforcement learning: training on distant goals reduces the mutual information between actions and goals, causing policies to become "goal-blind." To fix this, the authors introduce Sobolev Bootstrapping, a method that distills the sensitivity structure (input Jacobians) of short-horizon teacher policies into long-horizon students, significantly boosting performance in complex manipulation tasks.

What Problem Does This Paper Address?

In goal-conditioned reinforcement learning (GCRL), agents learn to reach any state by being "prompted" with a goal. A standard technique is hindsight relabeling, where failed trajectories are repurposed by treating the states actually reached as the intended goals.

Conventionally, the "curse of horizon" refers to the difficulty of learning long-term values due to noisy estimates or bias accumulation. However, these authors identify a distinct informational challenge. As the training goal gets further away in time, it provides less and less information about what the immediate next action should be. For example, if you are told to go to a city 100 miles away, your first few steps are less constrained than if you are told to reach a door 5 feet away.

The authors show that when policies are trained on these distant, low-information goals, they eventually learn to ignore the goal input entirely, focusing only on the current state. This leads to a catastrophic failure: the policy becomes unable to reach even nearby subgoals because it has lost its "sensitivity" to the goal.

Methods

The authors tackle this problem through both theoretical analysis and a new algorithmic intervention:

  1. Information-Theoretic Analysis: They prove that the conditional mutual information between actions and goals, $I(A; G | S)$, monotonically decreases as the relabeling horizon increases.
  2. Sensitivity Analysis via Jacobians: To measure what a neural network policy "pays attention to," they use the input Jacobian—the gradient of the action with respect to the state or goal. A small goal Jacobian indicates the policy is locally invariant to the goal.
  3. Sobolev Bootstrapping ($S^2AW$): They propose a modification to policy bootstrapping. In standard bootstrapping, a "student" policy (trained on long horizons) tries to match the action distribution of a "teacher" policy (trained on short, high-information horizons). Sobolev Bootstrapping goes a step further by forcing the student to also match the teacher's state Jacobians.

By distilling the sensitivity structure of the teacher, the student learns not just what to do, but what to pay attention to at each stage of a task.

Datasets

The researchers evaluated their methods on the OGBench suite, which contains 16 diverse offline environments:

  • Locomotion: PointMaze, AntMaze, and HumanoidMaze (ranging from medium to "giant" scales).
  • Combinatorial Manipulation: Tasks involving picking and placing multiple cubes (cube-single to cube-quadruple).
  • Complex Environments: "Scene" (a kitchen-like environment) and "Puzzle" (discrete tile-sliding tasks).

Key Insights and Ideas

  • Goal Sensitivity Collapse: Behavioral Cloning (BC) is particularly vulnerable to the informational curse. As shown in Figure 1, success rates plummet as the training horizon increases, even when the agent is only asked to reach nearby subgoals at test time. Training on distant goals harms nearby capabilities
  • The "State-Only" Shortcut: Neural networks are lazy. If the goal is distant and uninformative, the network finds it easier to ignore the goal and overfit to state features. This results in a global shift in sensitivity away from the goal (see Figure 2). Increasing training horizon decreases goal sensitivity
  • Jacobian Distillation as Regularization: Interestingly, matching only the state Jacobian of a short-horizon teacher actually helps recover goal sensitivity. It prevents the network from converging to a "state-only" solution.
  • Dynamic Attention: In multi-object tasks, Sobolev Bootstrapping allows the policy to dynamically shift its attention to the relevant object (the "focal cube") at each stage of the task, whereas standard methods exhibit "diffuse" sensitivity and fail early.
Figure 1: Training on more distant relabeled goals harms nearby goal-reaching capabilities. We record the success rates across policies trained on a geometric distribution of relabeling goal offsets with discount γ, where γ = 1 corresponds to a uniform distribution over future states. At evaluation, policies are given optimal Hplan-step subgoals from oracle planners. Success rates are averaged over the last three checkpoints, three seeds, and four environments: cube-double, cube-triple, cube-quadruple, and scene.
Figure 2: Increasing training goal horizon globally decreases policy goal sensitivity. We measure the proportion of the input Jacobian norm corresponding to the goal at fixed state-goal offsets, plotted with log2 scaling in cube-triple. Results are averaged over the last three training checkpoints, three seeds, and 1024 randomly sampled state-goal pairs per horizon, with additional environments in Figure 8.

Key Results

  • Massive Gains in Manipulation: On the cube-triple task, the baseline SAW algorithm achieved only a 4% success rate. Adding Sobolev distillation ($S^2AW$) boosted this to 30%. In cube-double, performance jumped from 40% to 57%.
  • Robustness to Distractors: $S^2AW$ proved much better at ignoring "distractor" objects. In environments with multiple cubes where only one needed to be moved, $S^2AW$ significantly outperformed all other baselines (see Table 2 in the paper).
  • Recovery of Goal Sensitivity: As shown in Figure 4, $S^2AW$ maintains high goal sensitivity across all goal offsets, whereas the baseline SAW policy's sensitivity collapses toward zero. Sobolev bootstrapping recovers goal sensitivity
  • Visualizing "Attention": Rollout visualizations (Figure 3) show that $S^2AW$ allocates its Jacobian norm (attention) to the specific cube it is currently interacting with, mirroring an optimal scripted trajectory. Visualizing attention during a triple pick-and-place
Sobolev bootstrapping recovers goal sensitivity
Visualizing attention during a triple pick-and-place

Limitations and Future Work

  • Computational Cost: Computing and backpropagating through Jacobians is computationally expensive, increasing training time (e.g., from 0.46 to 1.75 hours per million iterations in the giant humanoid maze).
  • Local vs. Global: Jacobians only capture local sensitivity. They are useful probes but not perfect estimators of the global mutual information the authors theorize about.
  • Horizon Selection: The "optimal" distribution of training horizons remains an open question. While Sobolev bootstrapping helps, finding ways to automatically balance short- and long-horizon information without explicit distillation is a promising future direction.

Summary updated

Abstract

The difficulty of learning goal-reaching policies is often attributed to a "curse of horizon" that manifests as bias accumulation in temporal-difference backups and noisy advantage estimates. In this work, we identify an additional informational curse of horizon in goal-conditioned policy learning, where increasing the goal relabeling horizon can significantly reduce policy generalization and performance. Through a series of controlled experiments with oracle planners, we decouple the goal horizons sampled during training from those that the policy is asked to reach at test time. Even when evaluated only on a sequence of nearby subgoals, goal-conditioned behavioral cloning (BC) policies suffer from severe, training horizon-dependent performance degradation that is mitigated by reinforcement learning (RL) objectives. We explain this phenomenon as a horizon-dependent decrease in the conditional mutual information between actions and hindsight-relabeled goals, and find empirically that both BC and RL policies trained on longer-horizon goals exhibit a shift in sensitivity from goal to state information, as measured by the policy's input Jacobians. Motivated by this observation, we find that distilling the input Jacobians of short-horizon policies into long-horizon policies yields significant performance gains, especially in combinatorial manipulation tasks. Taken together, our results highlight goal relabeling horizon as an important consideration when learning generalist policies from offline data.
BibTeX
@misc{zhou2026informational,
  author = {John L. Zhou and Yuxuan Dong and Jonathan C. Kao},
  title = {{An Informational Curse of Horizon in Goal-Conditioned Policy Learning}},
  year = {2026},
  eprint = {2610.09247},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.09247},
}
Figure