A common way to control a robot with reinforcement learning is to split the controller in two. A slow "manager" chooses what should happen next: a subgoal, or a skill with its settings, such as "push the box 10 cm to the left." A fast "worker" turns that choice into motor commands. Both are usually trained in a physics simulator and then moved to the real robot. The two levels meet at the subgoal, and that meeting point quietly assumes the real worker can reach whatever the simulated worker could reach. On hardware this assumption breaks. Motor delay, gear slack and friction make some subgoals unreachable. The manager keeps proposing them anyway, and the task fails even when each level is fine on its own.
We propose to measure and fix the reality gap (the performance lost when a simulator-trained policy runs on a real robot) at this interface rather than inside either policy. The steps are as follows. Freeze a worker trained in simulation. Spend a short real-robot budget watching which subgoals it actually reaches from which states. Learn a correction to the simulator's estimate of subgoal feasibility. Then re-train the manager in simulation under the corrected constraint. Because the correction describes what the robot's body can do rather than what one task needs, a single real-data collection should serve many downstream tasks. Methods that fine-tune the manager on real task reward cannot offer that reuse.
Core Claim
The claim has two parts. The first is a diagnostic. When a hierarchical policy fails on hardware, each failure can be split into two kinds. In a "subgoal miss," the worker did not reach the commanded subgoal. In a "subgoal success, task failure," the subgoal was reached but the world, for example an object, behaved differently than in simulation. The literature disagrees on which kind dominates. Hierarchical RL for quadruped soccer shooting kept a domain-randomized low-level kicking controller fixed and found the gap in the high-level planner, because ball compliance was not modeled. Aljalbout et al. and the bi-manual block assembly work instead trace failures to low-level actuator behavior. Our diagnostic makes this an empirical question for each task.
The second part is a method for the subgoal-miss share. Let $F_{sim}(s,g)$ be the probability, estimated in simulation, that the frozen worker reaches subgoal $g$ from state $s$ within the manager's decision interval. We learn a real-world correction:
$$\operatorname{logit} F_{real}(s,g) = \operatorname{logit} F_{sim}(s,g) + h_\phi(s,g),$$
Here $h_\phi$ is a small network fit by logistic regression on real reach/miss outcomes. Writing the correction as a residual on the simulator's estimate lets a few hundred real trials adjust it instead of learning it from scratch. The manager is then re-trained in simulation to maximize task return subject to $F_{real}(s,g) \ge \delta$, where $\delta$ is a feasibility threshold. The constraint is enforced with a Lagrange multiplier, the same way DIPPER uses the worker's value function to penalize unreachable subgoals. DIPPER supplies the mechanism. It learns its feasibility signal entirely in simulation, so it inherits exactly the mismatch we target. We keep its constraint and change where the constraint's numbers come from.
What would be new is threefold. The object adapted to reality is the level-to-level interface, not a policy. The real data needs no task reward. And the corrected feasibility model is meant to transfer across tasks that share a worker.
Research Program
The action structure follows the parameterized-primitive decomposition of ERLAP. The manager picks a discrete primitive type, such as approach, push, pivot or retreat, together with continuous parameters: a target pose change and a speed. ERLAP ran only in simulation, and abrupt primitive switches are exactly what Aljalbout et al. warn against on hardware. Our worker therefore executes every primitive through multi-step delta joint-velocity commands. These are small changes added to the previous velocity command, and they smooth motion. This is the action space Aljalbout et al. found transferred best, reaching 96% real pushing success while absolute joint-position control collapsed from 100% in simulation to 6% on the robot. The feasibility correction $h_\phi$ is conditioned on primitive type, so a push and a free-space approach can have different real-world reach envelopes.
The worker is trained once in simulation with domain randomization, meaning physical parameters such as friction, mass and latency are sampled from ranges during training. It is then frozen. Real-world interface data comes from the manager's own proposal distribution, deliberately oversampled near the simulated feasibility boundary where corrections matter most. Speed and workspace limits are enforced by the worker's low-level controller throughout. No task reward is recorded, only whether each subgoal was reached and the tracking error. The tracking error follows the normalized tracking error of Aljalbout et al., applied at the subgoal level rather than per command.
The analysis side builds the failure split described above for every task and robot. We will report the fraction of real failures that are subgoal misses, before and after correction. That fraction bounds what interface correction can ever deliver on a task. We expect it to be large for actuator-dominated tasks, such as fast pushes near the workspace edge or reorientation against a wall. We expect it to be small where object dynamics dominate, as in the soccer setting. We will report both regimes rather than choose the flattering one.
Baselines
The naive reference is zero-shot transfer: the hierarchical policy trained in simulation with domain randomization and deployed with no real data. It shows how large the gap is to begin with. The published analogue is the 40.6% zero-shot shooting accuracy reported by Sun et al.. That number comes from a different robot and task, so it serves as context only, not as a target.
The strongest competitor is real-world fine-tuning of the manager on task reward, as in Sun et al. They fine-tuned the high-level planner with REDQ, an off-policy method that trains many value estimates at once for sample efficiency, on 1,386 real kicks (about 23 minutes), and raised accuracy from 40.6% to 80.8%. This is the strongest competitor because it is the only method in the file that shows a large real-world gain for a hierarchy with a comparable data budget. We will run it with the same frozen worker, the same number of real transitions and the same hyperparameter search budget. We beat it if we match its success on the task where data was collected while using equal or fewer real transitions. The more important condition is on held-out tasks: there, our reused correction must exceed zero-shot success by a clear margin, while fine-tuning the manager would need fresh task-reward data.
The second competitor is simulation-only feasibility regularization, meaning DIPPER's constraint with $F_{sim}$ and no correction. It isolates the value of real grounding. DIPPER reports improvements of up to 35–40% over HAC and SAGA on Push and Franka Kitchen. No absolute reported success on Franka Kitchen was found in the file. We will re-run it ourselves under our budget.
The third competitor is a flat policy with multi-step delta joint velocities and domain randomization, following Aljalbout et al. It tests whether hierarchy is needed at all. It gets the same simulator steps and the same real-data budget for fine-tuning.
Two ablations isolate the key ingredient. One replaces the learned, state-dependent $h_\phi$ with a single global offset: uniformly shrinking every subgoal by a tuned factor. The other fits $F_{real}$ from real data alone, with no simulator estimate as a starting point.
Why not just…?
Why not just fine-tune the manager on real task reward, as the soccer work did? This is the real threat, and on a single task it may match or beat us. The soccer result shows that 23 minutes of real data can double accuracy. Our argument is not that manager fine-tuning is weak. It is that manager fine-tuning must be repeated for every task and mixes actuator error with object-dynamics error. The held-out-task experiment settles this. If a correction learned on one set of tasks does not improve new tasks sharing the same worker, the reuse argument fails, and the honest conclusion is that per-task manager fine-tuning is the better tool.
Why not just shrink the subgoal space conservatively? If real infeasibility were simply "the robot reaches a bit less far," a single tuned shrink factor would do everything our model does. We expect the real gap to depend on state and primitive, for instance slower speeds near joint limits or worse tracking during contact, so a uniform shrink should throw away reachable subgoals in some regions and still admit unreachable ones in others. The global-offset ablation tests this directly. If it comes within a few points of $h_\phi$ in success rate, we will report that the learned correction is unnecessary.
Why not just adapt the worker instead, for example with a dual-action domain-adaptation method? Dual Action Policy shows that giving a flat policy separate actions for task execution and for domain alignment can double the return of DARC-style baselines with as few as 1,000 target samples. It was tested only on simulated walkers with distorted bodies, not on hardware. Adapting the worker changes its behavior on every task and gives up the frozen, safety-checked controller. Adapting the interface leaves the worker untouched. We will include worker-level adaptation with the same real budget as an extra comparison. If it removes most subgoal misses, the diagnostic will show it, and the interface correction becomes redundant.
Why not just use a flat velocity-controlled policy with heavy randomization? Aljalbout et al.'s 96% came from single-stage pushing. The file offers no evidence that flat policies handle multi-stage real tasks at that level. The flat bi-manual policies needed billions of simulator steps and still plateaued at 50–65% real success (bi-manual attachment). The flat baseline on our multi-stage tasks measures this directly.
Proposed Experiments
The first stage is controlled sim-to-"real." The "real" system is a second simulator with unmodeled actuator latency, gear slack and altered friction. It runs the Push and Franka Kitchen tasks used by DIPPER. Here subgoal reachability can be measured exactly, so we can verify that $h_\phi$ recovers true reachability and sweep the real-data budget from about 200 to 2,000 subgoal trials.
The second stage is hardware, on a Franka Emika Panda. We use the Aljalbout et al. pushing setup, extended to chained multi-push and wall-assisted reorientation tasks modeled on the domains of the pre-/post-contact decomposition work. That work reports 87% zero-shot real success with a hand-set contact switch, which serves as a reference for what careful single-task engineering achieves. Interface data is collected on one group of tasks and reused on held-out tasks.
The metrics are real task success, the fraction of subgoals reached, the subgoal-miss share of failures, real transitions consumed, and safety events such as torque-limit hits and emergency stops.
There are three decisive checks. The first is whether the subgoal-miss share falls after correction while simulation-only regularization leaves it unchanged. The second is whether, on held-out tasks, the reused correction beats zero-shot transfer by a margin comparable to what manager fine-tuning achieves with fresh data. The third is whether the state-dependent correction beats the uniform shrink. If the first holds but the second fails, the contribution narrows to a diagnostic and a single-task method. If the third fails, the simpler shrink is the right recommendation, and we would say so.
References
- Hierarchical Reinforcement Learning for Precise Soccer Shooting Skills using a Quadrupedal Robot
- The Role of Action Abstractions in Robot Manipulation Learning and Sim-to-Real Transfer
- Bi-Manual Block Assembly via Sim-to-Real Reinforcement Learning
- Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
- Evolutionary Reinforcement Learning with Parameterized Action Primitives for Diverse Manipulation Tasks
- Dual Action Policy for Robust Sim-to-Real Reinforcement Learning
- Bi-Manual Manipulation and Attachment via Sim-to-Real Reinforcement Learning
- Pre-and Post-Contact Policy Decomposition for Non-Prehensile Manipulation with Zero-Shot Sim-To-Real Transfer