Offline agents that must chain many behaviors, such as opening a drawer, moving a cube into it, and pressing a button, still fail far more often than agents that only need to travel far. A common recipe is advantage-weighted imitation: copy the actions in a logged dataset, but give more weight to actions a learned value function says beat the average, and nearly zero weight to the rest. Hierarchical versions of this recipe add a "manager" that picks an intermediate target state and a "worker" that executes low-level actions to reach it. They still pick those targets in an arbitrary way, usually the state a fixed number of steps ahead. Humans can cheaply say where a task changes ("the drawer is now open," "cube 2 is now on the target"). The usual objection is that hand-written task structure is brittle and imposes suboptimal paths.
This note proposes to use cheap human annotations only to decide where to cut logged trajectories into subgoals, and to let advantage weighting decide which of those cuts to follow. The human supplies a handful of yes/no stage tests computed from the state. Each dataset trajectory is split wherever one of these tests flips value. The manager learns to imitate, with advantage weights, the real dataset state at the next flip. The claim is that event-aligned subgoals drawn from real data beat fixed-stride subgoals on compositional tasks. The more important claim is that this design degrades gracefully when the human annotation is wrong, incomplete, or in the wrong order. Rigidly following a human plan, the failure mode critics of manual decomposition point to, does not degrade gracefully.
Core Claim
Three groups of prior work frame the opportunity, and each plays a different role.
The first group supplies the method. HIQL trains one goal-conditioned value function and extracts both a manager and a worker by advantage-weighted regression, with subgoals defined as the state $k$ steps ahead. SAW keeps advantage-weighted bootstrapping toward real dataset waypoints but flattens the hierarchy away at test time.
The second group motivates the proposal. On the OGBench Scene environment, where tasks require up to eight sequential behaviors, CoGHP reports 78% success against 63% for SAW and 38% for HIQL. That gap suggests the trouble on multi-stage tasks lies in how subgoals are placed in time, not in the weighting rule. CoGHP closes much of the gap by predicting a whole chain of fixed-interval subgoals with a sequence model, not by changing where subgoals fall.
The third group constrains the design. GHRL proves that a known decomposition turns exponential exploration cost into linear cost. It does so only in tabular online mazes, and only with a rigid, fully correct structure map. Skill Q-Network argues that rigid manual decompositions become liabilities when conditions shift.
Our claim sits between these. If human stage tests set the subgoal boundaries, but the manager is an advantage-weighted imitator of whatever boundary the logged trajectory actually reached next, then the annotation only proposes candidate cuts and the value function adjudicates among them. Wrong orderings are harmless because the regression target is the boundary the data actually reached, not the human's planned next stage. Missing stages fall back to a fixed stride, which recovers HIQL. Spurious stages add extra cuts whose advantage is close to zero.
Unlike adInfoHRL (Advantage-Weighted Information Maximization), which discovers discrete options online by advantage-weighted clustering, our subgoals are concrete, checkable states. Unlike MaMiC, which finds stage boundaries from spikes in a hand-built dense reward and then trains a fixed curriculum, we need no dense reward and never commit to a single stage order.
Method
The human writes $m$ binary predicates $c_1,\dots,c_m$ over states. In OGBench Scene these are tests like "drawer open" or "button lit." In Cube they are "cube $i$ at its target." In FrankaKitchen they are the four subtask completion tests. For a dataset trajectory, let $b(t)$ be the first time after $t$ at which any predicate changes value, capped at $t + k_{\max}$. The cap means that with no predicates the method reduces exactly to HIQL with stride $k_{\max}$.
We train a goal-conditioned value $V(s,g)$ with action-free implicit Q-learning (IQL), as in HIQL. The manager $\pi_h$ outputs a subgoal representation $z$, using the value network's internal encoding $\phi$ as in HIQL. It is trained by
$$\max_{\pi_h}\ \mathbb{E}_{(s_t,g)\sim\mathcal{D}}\Big[\exp\!\big(\beta_h\,(V(s_{b(t)},g)-V(s_t,g))\big)\,\log \pi_h\big(\phi(s_{b(t)})\mid s_t,g\big)\Big],$$
where $\beta_h>0$ sets how sharply the manager prefers boundaries that make progress toward the final goal $g$. The worker $\pi_\ell$ is trained the same way at the action level:
$$\max_{\pi_\ell}\ \mathbb{E}\Big[\exp\!\big(\beta_\ell\,A(s_t,a_t,s_{b(t)})\big)\log \pi_\ell\big(a_t\mid s_t,\phi(s_{b(t)})\big)\Big],$$
where $A$ is the one-step advantage toward the subgoal, $r + \gamma V(s_{t+1},s_{b(t)}) - V(s_t,s_{b(t)})$.
The only new ingredient relative to HIQL is the boundary function $b(t)$. That is deliberate. It makes the decisive ablation clean, because swapping human-aligned cuts for random cuts with the same count isolates the value of the annotation itself. The temperatures will be set by the effective-sample-size rule used in PAWS, which addresses the known sensitivity of these methods to $\beta$.
Because every manager target is a real dataset state at which a predicate genuinely flipped, the manager is trained only on reachable subgoals. This addresses, offline and without a generative image model, the infeasibility problem that Strict Subgoal Execution handles with online stop-on-failure and that GHIL-Glue handles with a learned filter over generated images.
Research Program
The program has three parts.
The first is the method above, implemented on the HIQL codebase structure so that every component except $b(t)$ is shared with the strongest hierarchical baselines.
The second, and the heart of the study, is an annotation-corruption protocol. It turns the brittleness critique of manual decomposition into a measurable curve. We will corrupt the predicate set in four ways, each at graded severity. We will delete a fraction of the predicates, which produces missing stages. We will add predicates that threshold irrelevant state coordinates, which produces spurious stages. We will jitter flip times by up to $j$ steps. Finally, for methods that consume a planned order, we will permute the human's stated stage order.
The third part is an analysis of why event-aligned cuts help. Following HIQL's signal-to-noise argument, we will measure the gap between the manager-level advantages of good and bad subgoals relative to the spread of value estimates across seeds. We will compare that gap under human-aligned, fixed-stride, and random-count-matched boundaries. If human cuts help, they should help by raising this gap at stage transitions, where fixed strides land mid-motion.
Baselines
The naive reference is goal-conditioned behavioral cloning, which copies all logged actions without values or hierarchy. It is trained on the same data with the same network. It establishes how much any value-aware method adds. PAWS reports roughly 48% for behavioral cloning against roughly 70% for its advantage-weighted method on Meta-World Door Open, a sign that this reference is not trivially beaten.
The flat value-aware reference is goal-conditioned IQL with advantage-weighted extraction, which HIQL reports at roughly 20–30% on AntMaze-Ultra against HIQL's 53%.
HIQL itself is the direct parent and the "same method, no human cuts" ablation. SAW is the strongest flat competitor.
CoGHP is the strongest existing method on our target question, because it is the best reported result on the multi-stage Scene task: 78% success, with SAW at 63% and HIQL at 38%, all reported in the CoGHP paper. Beating it means exceeding 78% on Scene under the same evaluation goals. Alternatively, it means matching it while degrading less under corruption, since CoGHP uses no annotation and is unaffected by it.
On FrankaKitchen, the strongest reported numbers in the file come from LAPO, which reports 62.42 on kitchen-mixed against 49.92 for IQL. ICQL reports a 16.4% average gain over the best baseline on Kitchen tasks but no absolute kitchen-mixed score in the file. For SAW and CoGHP on Kitchen, no reported result was found on Lacuna.
Two annotation-using baselines make the comparison fair to the manual-decomposition idea itself. The first is HIQL with the predicate vector appended to the state, which gives the network the same information without changing where subgoals fall. The second is hard chaining in the spirit of GHRL. A fixed human stage order is executed by a goal-conditioned worker trained with advantage-weighted regression, and each stage ends when its predicate flips.
Two ablations isolate our ingredients. The random-cut ablation uses the same number of boundaries per trajectory, placed at random. The uniform-manager ablation sets $\beta_h \to 0$, so the manager imitates the next boundary without advantage weighting.
All methods share the image or state encoder, the value-learning hyperparameters, eight seeds, and an equal tuning budget of the same number of sweep configurations per method. We will first reproduce the published HIQL, SAW, and CoGHP Scene numbers before reporting comparisons.
Why not just…?
Why not just append the stage predicates to the input of HIQL or SAW? This is the most serious threat. If it matches our method, the contribution is "give the network the predicates," and the boundary mechanism is unnecessary. The HIQL-plus-predicates baseline exists to settle this. Our hypothesis is that extra inputs do not fix the problem CoGHP identifies: a fixed stride places manager targets mid-motion, where value differences between good and bad subgoals are smallest. The signal-to-noise measurement will show directly whether the input-augmented model still suffers small manager advantages at stage transitions. If it does not, and it matches our success rates, we will report that the boundary placement adds nothing.
Why not just shape the reward with bonuses for each completed stage? Stage bonuses redefine the objective. Evidence in the file shows this misleads agents: Skill Q-Network documents agents trapped by optimism toward proximate shaped rewards. In our setting, a spurious or wrongly ordered stage becomes a false reward rather than a candidate cut the value function can ignore. We will include a shaped-reward goal-conditioned IQL run under the same corruption protocol. The prediction is that its performance falls with spurious predicates while ours stays near the uncorrupted level.
Why not just follow the human's stage plan with a good low-level policy? When the plan is right, this may be as good as or better than our method, and GHRL's theory says it should be very sample-efficient. The hard-chaining baseline tests exactly that. The case against it is the corruption curve. Under permuted orders or deleted stages, hard chaining has no way to recover, whereas our manager only ever regresses toward boundaries the data actually reached. If hard chaining wins even at moderate corruption, the robustness claim fails, and we will say so.
Why not just use CoGHP, which already reaches 78% on Scene without any annotation? CoGHP shows that a sequence model can learn long chains of fixed-interval subgoals. It needs no human input, and that is a real advantage. Our claim is narrower: a few human predicates, costing minutes to write, should help most where the data are least structured. On the play and explore splits, fixed-interval chains may not line up with any task stage. On the hardest Puzzle levels, the OGBench paper reports 0% success for all tested methods. If CoGHP matches us there too, annotation is not worth its cost on these benchmarks.
Proposed Experiments
The primary suite is OGBench manipulation: Scene, Cube (single to triple), and Puzzle. These tasks have natural state-computable predicates, and the reported numbers above give clear anchors. We will use the standard five evaluation goals per environment and report success rate.
FrankaKitchen from D4RL (complete, partial, mixed) is the second suite, scored on normalized return. Its four subtask tests are the canonical human decomposition, and the mixed split contains subtasks in many orders. That makes it the natural test of the order-permutation claim.
As a negative control, we will run OGBench AntMaze-Giant with room-membership predicates. SAW reports 73% there and CoGHP 78%. We expect little gain, since navigation stages are less semantically distinct, and finding a large gain would suggest the benefit comes from something other than stage alignment.
Four results decide the proposal.
The first is whether human-aligned cuts beat random cuts with the same count. If not, the annotation carries no information beyond variable stride length.
The second is whether the method beats CoGHP's reported 78% on Scene, or opens a clear margin on Puzzle and the low-quality data splits.
The third is whether, along the corruption curve, our success rate stays well above hard chaining and shaped rewards as predicates are deleted, added, jittered, or reordered. With all predicates deleted, it should fall back to HIQL's level rather than below it.
The fourth is whether the uniform-manager ablation loses specifically under spurious and reordered annotations. That would be direct evidence that advantage weighting is what lets the agent ignore bad human advice.
A positive result on all four would show that manual decomposition is useful offline when treated as a proposal of where to cut rather than as a plan to obey. A failure on the first or third would be equally informative. It would mean the brittleness critique of manual decomposition holds even with advantage weighting on top.
References
- HIQL: Offline Goal-Conditioned RL with Latent States as Actions
- Flattening Hierarchies with Policy Bootstrapping
- Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
- Efficient Exploitation of Hierarchical Structure in Sparse Reward Reinforcement Learning
- Skill Q-Network: Learning Adaptive Skill Ensemble for Mapless Navigation in Unknown Environments
- Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization
- MaMiC: Macro and Micro Curriculum for Robotic Reinforcement Learning
- PAWS: Preference Learning with Advantage-Weighted Segments
- Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning
- GHIL-Glue: Hierarchical Control with Filtered Subgoal Images
- LAPO: Latent-Variable Advantage-Weighted Policy Optimization for Offline Reinforcement Learning
- In-Context Compositional Q-Learning for Offline Reinforcement Learning
- OGBench: Benchmarking Offline Goal-Conditioned RL
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning.