A hierarchical robot policy splits control into two parts. A "manager" picks intermediate targets, called subgoals, every few dozen steps, and a "worker" issues motor commands to reach them. This split makes long, multi-stage tasks learnable in simulation. It also has a known weakness: the manager only sees the world through the worker. Every time the worker changes, because it is still learning or because the robot has moved from simulation to real hardware with different friction and motor lag, the manager's picture of what happens when it requests a subgoal becomes wrong. Most methods for stabilizing hierarchies assume the physics never changes. Most sim-to-real methods assume the policy is a single flat network. On a real robot both kinds of change arrive together, and real interaction is the scarcest resource.
We propose to make one learned object the only interface between the two levels: an outcome model that predicts where the current worker actually ends up when asked to reach subgoal $g$ from state $s$. The manager learns from imagined transitions drawn from this model rather than from a stale replay buffer. When the robot moves to the real world, we correct only this low-dimensional model, using a small number of real subgoal executions, including the failed ones. The bet is that the reality gap is cheap to fix at the level of "what does the worker achieve" and expensive to fix at the level of "what does each motor command do." Fixing it at the outcome level should therefore recover a sim-trained hierarchy with far fewer real transitions than grounding the primitive dynamics.
Core Claim
The manager experiences a transition law that mixes physics and the worker, $P_H(s' \mid s, g) = \sum_a P(s' \mid s, a)\, \pi_L(a \mid s, g)$. Here $P$ is the physical one-step dynamics and $\pi_L$ is the worker policy, and over a subgoal window of $c$ steps the composition repeats $c$ times. Two lines of work each treat half of this mixture. Hierarchical stabilization methods hold $P$ fixed and tame changes in $\pi_L$. Sim-to-real grounding holds the policy flat and corrects $P$. Our claim is that both shifts can be absorbed by tracking $P_H$ directly. We do this with a sim-pretrained outcome model $W_\theta(s' \mid s, g)$ plus a small residual $r_\psi$ fitted on real executions:
$$W_{\text{real}}(s' \mid s, g) = W_\theta(s' \mid s, g) \circ r_\psi(s, g),$$
where $r_\psi$ shifts and rescales the predicted outcome distribution. In practice $W_\theta$ outputs a Gaussian or mixture over the worker-controlled coordinates of $s'$, and $r_\psi$ adjusts its mean and spread. The manager is trained Dyna-style, meaning on short imagined rollouts, using transitions sampled from $W_{\text{real}}$. Subgoal feasibility is scored by the grounded probability of landing near $g$, not by a simulator-trained worker value function.
Three things are new relative to the closest work. First, grounding happens at the subgoal level rather than the primitive-action level. Second, a single model handles both the reality gap and the secondary drift that occurs when the worker itself keeps adapting on hardware. Third, failed worker executions become ordinary training data for the outcome model rather than noise to be thrown away.
Relation to Prior Work
Three papers motivate the problem by showing that the manager's inputs go stale.
- Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies documents that a still-learning worker gives the manager inconsistent feedback.
- Data-efficient Hindsight Off-policy Option Learning (HO2) shows that replay transitions collected under an old worker become invalid. Its answer is to marginalize over all options in hindsight with dynamic programming, inside simulation only.
- Sample Complexity of Goal-Conditioned Hierarchical Reinforcement Learning proves that hierarchical sample cost is bounded by the harder of the two levels. Its stationary variant (SHQL) updates the manager only on successful worker completions, and it is tabular only.
These two papers pull in opposite directions. HO2 says to use every trajectory. SHQL says to discard failures to restore stationarity. Our design resolves the conflict on the side of keeping data. A failure is not noise once it is read as a sample of where the worker actually went, which is exactly what $W$ is trained to predict. This matters most on hardware, where discarding real failures wastes the most expensive data.
The method borrows its machinery from two sources.
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer (RGAT) trains an action-transforming policy so that simulated transitions match "real" ones. It does this only for flat policies, with a perturbed simulator standing in for reality. We reuse its evaluation protocol and its idea of learning the correction, but we move the correction from primitive actions to subgoal outcomes.
- Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach (DIPPER) trains the manager on a fixed preference dataset of about 10,000 comparisons. It penalizes infeasible subgoals using the worker's value function. That fixed dataset and simulator-trained value are precisely what break under physical change. We replace the feasibility term with a grounded outcome probability that a few real executions can update.
Two papers constrain the design. HiWET shows that a hierarchy can transfer zero-shot to a humanoid, but only by training the levels separately around a frozen kinematic prior. Its ablation also shows that learned residual corrections matter: hand tracking error doubled from 12.4 mm to 25.2 mm without them. So we do not freeze the worker by default. Flat world-model methods such as MOTO and Simulation Distillation adapt full-dynamics models. MOTO in particular must penalize model exploitation through ensemble uncertainty. Our outcome model predicts only subgoal-scale endpoints, which limits the space a manager can exploit, but we inherit the ensemble penalty from MOTO.
Research Program
Stage one: sim pretraining. We train a goal-conditioned hierarchy in simulation in the DIPPER style, with a SAC worker. Alongside it we train $W_\theta$ on worker executions from recent policy versions. In simulation, stale data is cheap to fix: we periodically re-execute stored $(s, g)$ pairs with the current worker. This keeps $W_\theta$ synchronized with $\pi_L$, and the manager is trained on samples from $W_\theta$ rather than raw buffer transitions. This stage alone is a test of whether the outcome interface stabilizes joint training, before any reality gap is introduced.
Stage two: target-domain adaptation under a fixed real budget. We collect real executions of manager-proposed subgoals and fit the residual $r_\psi$ by maximum likelihood on observed endpoints. Failures are kept. We then update the manager on imagined transitions from $W_{\text{real}}$, restricting imagination to regions where an ensemble of residuals agrees.
We study two worker regimes:
- Frozen worker. Only the outcome model and manager adapt.
- Adapting worker. The worker also fine-tunes on real data. Here $r_\psi$ is refit on a sliding window so the manager tracks the worker's drift, which is the coupled physical-plus-hierarchical non-stationarity that no prior work in the file has formulated.
Analysis. We measure the dimensionality and smoothness of the outcome correction against the primitive dynamics correction. This tests the core hypothesis that the outcome gap is simpler to learn. We also measure how closely $W$ tracks true worker outcomes over training, which serves as a direct readout of non-stationarity.
Baselines
Naive reference: sim-trained hierarchy deployed unchanged. This is the DIPPER-style hierarchy from stage one run directly in the target domain. It bounds how much the reality gap hurts.
Upper reference: hierarchy trained entirely in the target domain. This is the same architecture with unrestricted target interaction, following RGAT's "policy trained directly on the modified environment" reference. Results are reported as the fraction of the gap between the naive and upper references that each method closes.
Strongest sim-to-real competitor: RGAT applied to the worker's primitive actions. We train RGAT's action transformer on the same real-transition budget, re-train or fine-tune the hierarchy in the grounded simulator, and deploy. It is the strongest competitor because, if primitive dynamics are grounded well, $P_H$ is automatically right as well. No numerical episodic reward is reported for RGAT on Lacuna. It is reported to match target-trained policies on InvertedPendulum, Hopper and HalfCheetah, where classical GAT failed. Beating it means higher target success at equal real budgets. The gap should be largest at the smallest budgets and for longer subgoal horizons $c$, where per-step errors compound.
Strongest hierarchical competitor: DIPPER fine-tuned on target data. The worker value function is re-estimated on the same budget. Its reported result is a success-rate improvement of up to 40% over HAC, SAGA and PIPER on the robotic arm suite and Franka Kitchen, measured in simulation without any reality gap. We also run an SHQL-style competitor that updates the manager only on successful worker completions.
Flat model-based competitor. On Franka Kitchen we include MOTO-style flat world-model fine-tuning. MOTO reports 100% success on the Mixed task and 90.5% on the Partial task from pixels, against DreamerV2's 13.5% on Partial, all in-sim and without a reality gap.
Fairness and ablations. All methods share the same simulator pretraining compute, the same real-transition budgets, and the same per-method hyperparameter search budget, with five or more seeds each. Four ablations isolate the ingredients:
- removing the residual, so the sim-only $W_\theta$ is used on target;
- dropping failed executions from residual fitting;
- training the manager on raw real buffer transitions instead of $W$ samples;
- turning off sliding-window refitting in the adapting-worker regime.
Why not just…?
Why not just ground the simulator at the primitive level with RGAT and retrain the whole hierarchy? This is the real threat. If it matches our method at equal real budgets, our contribution shrinks to an analysis result. Primitive grounding does get $c$ times more labeled transitions per real subgoal execution. Our hypothesis is that it must learn a higher-dimensional correction whose small errors compound across each $c$-step window. The outcome residual learns only the endpoint shift the manager actually uses. The budget sweep crossed with varying $c$ settles this. If RGAT-on-worker wins at every budget and horizon, we will report that grounding belongs at the primitive level.
Why not just widen domain randomization over the whole hierarchy, as DrEureka does? DrEureka automates randomization ranges and transfers flat policies zero-shot. RGAT's authors argue that wide randomization yields overly conservative policies in precision tasks. A hierarchy is especially exposed, because a conservative worker shrinks the reachable set the manager plans over. We include a randomized-hierarchy baseline at matched pretraining compute. If it closes the gap on contact-rich Kitchen tasks with zero real data, that is a strong negative result for our approach and we will report it.
Why not just freeze the sim-trained worker and fine-tune the manager on real successes? This is SHQL's recipe carried over to the real robot. It removes hierarchical non-stationarity but throws away failures and cannot repair a worker broken by the reality gap. The frozen-worker arm of our method uses the same data but keeps the failures. The success-only ablation then isolates whether failures carry the information we claim.
Why not just collect demonstrations and imitate? HiLAM reports that flat imitation (BAKU) reaches less than half of HiLAM's 45% success on LIBERO-Long at 10% of demonstrations, showing that flat imitation is data-hungry on long tasks. Imitation also cannot adapt to wear or payload changes. We treat it as out of scope rather than a direct competitor.
Proposed Experiments
Sim-to-sim stage. Following RGAT's protocol, the target domain is a perturbed simulator with altered friction, joint damping, actuator latency and payload mass. Tasks are DIPPER's sparse-reward arm suite (maze navigation, pick-and-place, push) and Franka Kitchen. Methods are compared at real budgets spanning roughly one to one hundred thousand target transitions. The primary metric is task success rate as a function of real transitions consumed. Secondary metrics are outcome-prediction error of the manager's model over time and the fraction of manager subgoals reached on target.
Decisive check one: data efficiency at the outcome level. On the hardest perturbations, the outcome-grounded hierarchy should close at least half of the naive-to-oracle gap with an order of magnitude fewer target transitions than worker-level RGAT. The advantage should grow with subgoal horizon $c$.
Decisive check two: coupled drift. We stress-test the adapting-worker regime. If the claim holds, the raw-buffer manager's success should drop while the worker fine-tunes on target, and the sliding-window outcome model should prevent that drop.
Optional hardware stage. Contingent on the sim-to-sim results, we repeat the pick-and-place task on a physical 7-DoF arm with a budget of minutes of real interaction. The limit is in the spirit of the under-four-minute autotuning reported by Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids. No reported result was found on Lacuna for hierarchical RL under such a budget.
References
- Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies
- Data-efficient Hindsight Off-policy Option Learning
- Sample Complexity of Goal-Conditioned Hierarchical Reinforcement Learning
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
- HiWET: Hierarchical World-Frame End-Effector Tracking for Long-Horizon Humanoid Loco-Manipulation
- MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
- Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation
- DrEureka: Language Model Guided Sim-To-Real Transfer
- Hierarchical Latent Action Model
- Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids