Robots trained in simulation often fail on real hardware. A common remedy is to have a person watch the robot and take over the controls whenever it is about to fail, then use those takeovers as a learning signal. Current methods treat every takeover the same way: the action just before it gets a penalty. For long, multi-stage tasks run by a two-level controller, that is the wrong lesson. A high-level planner picks intermediate targets ("grasp the mug at its handle"), and a low-level controller moves the arm to reach each one. When a person takes over, the planner may have chosen a target that cannot be reached on real hardware, or the controller may have fumbled a target that was fine. We propose to tell these two cases apart from something the human already does. If the human's corrective motion still ends near the target the planner chose, the controller failed. If the human steers somewhere else, the planner failed. Each takeover then updates only the level that caused it.
The mechanism that makes this pay off is a real-world reachability model. This is a learned estimate of whether the low-level controller can reach a given target from a given state on the physical robot. It is trained on target segments the human let run to completion (positive examples) and on segments the human aborted and attributed to the controller (negative examples). The planner is then steered away from targets the model marks as unreachable in reality. This replaces the simulation-trained estimate that current hierarchical methods use for the same purpose, which is exactly the quantity the reality gap corrupts. The claim is that this routing of human signal reaches a target success rate on hardware with fewer interaction minutes than flat takeover-based fine-tuning. The advantage should appear specifically when the reality gap makes some simulator-feasible targets infeasible.
Where this sits in prior work
Three lines of work interpret takeovers and non-takeovers as feedback, and all three operate on flat policies. Expert Intervention Learning splits a trajectory into states the supervisor tolerated, states just before takeover, and the human's recovery, then turns these into constraints on an action-value function for a small racing car. RLIF assigns reward −1 at takeover and 0 otherwise, and trains with off-policy RL on prior data. MILE fits a differentiable model of when the human will intervene, and learns from non-intervention periods as well. These papers supply our signal and our main baseline. We differ in two ways. Each takeover is assigned to one of two time scales rather than to the last motor command. And learning starts from a simulation-trained policy rather than from scratch on hardware. MILE explicitly lists sim-to-real as future work, and RLIF's hardware runs learn directly on the robot. MEReQ does start from a prior policy and infers a residual reward from interventions. However, it improves the policy by running RL inside a simulator, so a contact-physics gap flows straight back into the update. We never optimize against simulator dynamics after transfer.
On the hierarchical side, DIPPER and its successor Hierarchical Preference Optimization supply the method we reuse. They train the planner on a fixed preference dataset, which avoids the instability of a planner chasing an improving controller. They also penalize targets that the low-level value function rates as unlikely to succeed. That penalty is the right idea, but the controller's value function is learned in simulation, and DIPPER is evaluated only in simulation using about 10,000 offline comparisons. We keep the penalty and replace its source with the real-world reachability model. On Credit Assignment in Hierarchical RL motivates the sample-efficiency argument. It shows that hierarchical backups act like multistep updates that skip over intermediate motor steps, so a planner-level error can be credited at the planner's time scale instead of being diluted across many motor steps. Its results are tabular, and we do not assume they carry over to continuous control. Testing that is part of the program.
Grounding methods constrain the design. RGAT learns an action transformer that bends simulator dynamics toward reality, but it is evaluated against a perturbed simulator rather than hardware. Its finding that composed dynamics models break under deep policies argues against our first instinct, which was to ground the low level with a learned simulator correction. We adapt the controller with real takeovers instead. The strategy-aware feedback work of Loftin et al. and Peng et al. warns that human silence is ambiguous: trainers withhold feedback strategically, and about 15% switch strategy mid-session. This is why we treat silence as a positive label only when the segment's completion can also be checked against sensors, rather than as endorsement by default.
Core Claim
The planner sends targets $g_k$ to the controller for segments $k = 1, 2, \dots$. A segment ends in one of three ways: the target is reached without takeover, the segment times out, or the human takes over. When the human takes over, they drive the robot to some state $s^{h}_k$ before handing back control. The attribution rule is
$$\ell_k = \begin{cases} \text{controller} & \text{if } d(s^{h}_k, g_k) < \epsilon \\ \text{planner} & \text{otherwise,} \end{cases}$$
where $d$ is a task-space distance and $\epsilon$ is the target tolerance the controller already uses.
The real-world reachability model $F(s, g)$ estimates the probability that the controller reaches $g$ from $s$ on hardware without takeover. Its training labels come from two sources. Segments that reached their target without takeover are positives. Controller-attributed takeovers and timeouts are negatives.
Planner-attributed takeovers carry a different meaning: the target was the wrong idea, regardless of whether it was reachable. They feed the planner in two ways. They act as a −1 reward at the planner's time scale. They also form a preference pair: the human's endpoint $s^h_k$ is preferred over $g_k$, which plugs directly into DIPPER's preference objective. The planner's feasibility penalty becomes $-\lambda \log F(s, g)$ in place of the simulation value estimate. Controller-attributed takeovers go to the controller as RLIF-style −1 rewards, and the human's corrective motion is added to its replay data.
The falsifiable prediction concerns the type of gap. If the reality gap only degrades motor tracking, simply fine-tuning the controller should match our method. If the gap makes some simulator-feasible targets infeasible, ours should keep improving after controller-only fine-tuning plateaus. An example is grasp poses that hold in simulation but slip on a real object.
Research Program
The first stage builds the pieces in a controlled setting where ground truth is known. We pretrain a DIPPER-style hierarchy in robosuite on multi-stage tasks: Pick-and-Place, Nut Assembly, and a Franka Kitchen-like sequence. The controller is a learned policy on top of operational-space control.
Following RGAT's protocol, the "real" world is a second simulator with perturbed parameters, and we inject gaps of three kinds. The first is tracking-only: actuator gain and latency change, but every target remains reachable. The second makes some targets infeasible: friction and object mass change so that a subset of grasp poses fail. The third mixes both. The human is played by scripted interveners in the style of RLIF's simulated experts. Some interveners correct toward the original target when the controller fails and redirect when the target is infeasible. Others are deliberately noisy: they redirect even on controller failures, react late, or change strategy partway through, to mimic the variability Peng et al. document. Because the injected gap is known, attribution accuracy can be measured directly.
The second stage tests whether the skip-connection argument survives deep function approximation. We compare planner learning curves when planner-attributed rewards are applied at the planner's time scale versus spread over the motor steps of the segment.
The third stage moves to a Franka arm on a two-stage task: grasp a peg from varied poses, then insert it at RLIF's 1.5 mm clearance. Here the gap is real, and humans intervene with a SpaceMouse as in MEReQ. We record whether operators' corrections actually follow the "same target versus different target" pattern our rule assumes. That human-behavior question is the main risk in the whole proposal.
Baselines
The do-nothing reference is the simulation-trained hierarchy deployed without adaptation. This measures how large the gap is. The best published zero-shot point in the file is ManiSkill2's PickCube: 91% success in simulation and 60% on a physical robot with no real-world fine-tuning (ManiSkill2). We expect our multi-stage task to transfer worse than that.
The strongest existing competitor is RLIF applied to a flat policy pretrained in the same simulator, then fine-tuned on hardware with takeover penalties. It is the strongest because it already turns takeovers into RL rewards, it outperformed DAgger-like methods by 2-3x on average in simulation, and on a real Franka it reached 100% peg-insertion success within 20 minutes over 6 interaction rounds (RLIF). That number comes from a single-stage task learned on hardware, so it is a reference scale rather than a directly comparable target. We will rerun RLIF ourselves on our task. Every method gets the same pretraining data and simulator budget, the same operators, the same cap on hardware minutes, and the same hyperparameter search budget, done in the surrogate setting.
MILE serves as a second interactive baseline, using its reported 80% success after 4 iterations on a physical WidowX (MILE) as a scale reference only. HG-DAgger is included as the supervised interactive-imitation reference.
Three ablations isolate the key ingredient. The first is uniform hierarchical RLIF, where every takeover penalizes both levels. The second is controller-only fine-tuning, where the planner is frozen. The third keeps attribution but uses DIPPER's simulation value function in the planner's feasibility penalty instead of the real-world reachability model. An explicit-labeling variant, in which the operator presses a button marking the failed level, gives an upper bound on attribution quality.
Beating RLIF means reaching 80% success on the two-stage hardware task in at least 30% fewer interaction minutes, or reaching higher success at an equal budget, across three seeds and at least two operators.
Why not just…?
Why not just fine-tune a flat policy with RLIF, as already shown on hardware? This is the baseline to beat, and on short single-stage tasks it may well win, since there is nothing to attribute. Our prediction is limited to multi-stage tasks with the target-infeasible gap. In those tasks, a flat −1 at the takeover step lands on motor actions that were not the cause, a credit-assignment problem de Vries et al. characterize in the tabular case. The surrogate experiments with each gap type will show whether this matters in practice.
Why not just freeze the planner and fine-tune only the controller, since the reality gap is "in the physics"? This is the most serious threat, and the prediction is built around it. If the gap is tracking-only, we expect controller-only fine-tuning to tie with our method, and we will report that as a negative result rather than tune around it. The case it cannot handle is a target that no controller can reach on the real object, such as a grasp pose that slips. There, only changing the planner's choices helps. The tracking-only versus target-infeasible comparison settles this directly.
Why not just ask the operator to say which level failed? It is simpler and probably more accurate, so it is included as an upper bound rather than dismissed. Its costs are an extra cognitive load during split-second takeovers, plus the label noise and strategy drift that Peng et al. report for explicit feedback. If button labels match our inferred attribution at similar operator effort, the contribution narrows to the real-world reachability model and the planner coupling, which the third ablation still tests independently.
Why not just widen domain randomization? Karnan et al. argue that broad randomization makes policies conservative on contact-rich tasks, and SURREAL left Nut-and-peg Assembly unsolved even with 64 actors (SURREAL). We will still pretrain every method with moderate randomization, so any gain we report is on top of it.
Proposed Experiments
The primary metric is success rate as a function of cumulative hardware interaction minutes and number of takeovers. The secondary metric is attribution accuracy against the known injected gap in the surrogate. We also report the calibration of the reachability model on held-out hardware segments.
The decisive check has three parts. In the target-infeasible surrogate condition, our method should exceed controller-only fine-tuning at the end of the budget. In the tracking-only condition, the two should tie. And replacing the real-world reachability model with the simulation value function should erase most of the gain.
The hardware-specific check is whether real operators' corrective endpoints separate cleanly under the distance rule. We will measure this against the button labels collected in the explicit-labeling arm. If the endpoints do not separate, the attribution premise fails on real people, and the surrogate gains will not transfer.
References
- Learning from Interventions: Human-robot interaction as both explicit and implicit feedback
- RLIF: Interactive Imitation Learning as Reinforcement Learning
- MILE: Model-Based Intervention Learning
- MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention
- Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
- On Credit Assignment in Hierarchical Reinforcement Learning
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- A Strategy-Aware Technique for Learning Behaviors from Discrete Human Feedback
- Learning something from nothing: Leveraging implicit human feedback strategies
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
- SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark