Recalibrating Subgoal Feasibility Across the Reality Gap

Many robot controllers are split in two. A slow "manager" picks intermediate targets, called subgoals, such as "move the gripper to this point" or "put the block here." A fast "worker" turns each subgoal into motor commands. The best recent methods for training such hierarchies teach the manager to avoid impossible subgoals by consulting the worker's own estimate of how likely it is to succeed. When the system moves from a physics simulator to a real robot, that estimate stays behind in simulation. Friction, actuator lag and torque limits change what the worker can actually reach. The manager keeps proposing targets that were reachable in the simulator but are not on hardware.

This note proposes to correct only that estimate, using a small number of real subgoal attempts chosen near the edge of what the worker can do. The manager is then retrained against the corrected estimate. The claim is that, under a reality gap, much of the performance loss in hierarchical controllers occurs at the manager–worker interface. If so, fixing the feasibility signal should recover more performance per real trial than grounding the whole simulator or fine-tuning the whole worker. This claim can fail, and the experiments are designed to show whether it does.

Core Claim

The mechanism builds on a bi-level method: one that trains the manager and worker as a nested optimization, with the manager as leader and the worker as follower. Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach (DIPPER) and its predecessor Hierarchical Preference Optimization take this approach. They train the manager with Direct Preference Optimization, which learns from pairwise comparisons of trajectories rather than from a reward, on a fixed offline dataset. They add a penalty, weighted by a Lagrange multiplier, whenever the worker's value function $V^{\pi L}(s,g)$ judges subgoal $g$ unreachable from state $s$.

These papers supply the method we reuse. They also motivate the gap: every evaluation is in MuJoCo, so the penalty is only as good as the simulator. Reinforced Grounded Action Transformation (RGAT) shows that transitions learned in simulation drift on a different "real" system unless actions are explicitly transformed. That result implies a simulation-trained $V^{\pi L}$ gives a biased sense of feasibility.

We replace the simulation value with a grounded reach probability:

$$\hat p(s,g)=\sigma\big(\alpha\,V_{\text{sim}}(s,g)+\beta+h_\psi(s,g)\big).$$

Here $\sigma$ is the logistic function and $V_{\text{sim}}$ is the frozen simulation worker value. The scalars $\alpha,\beta$ recalibrate that value, and $h_\psi$ is a small residual network. All three are fit by maximum likelihood on binary outcomes: did the real worker reach $g$ from $s$ within the manager's step budget?

The manager's constraint becomes $\hat p(s,g)\ge\tau$, where $\tau$ is a chosen reach-probability threshold. Real trials are selected actively. We probe subgoals where $\hat p$ is near $\tau$ and the ensemble over $\psi$ disagrees most, because errors far from the boundary do not change which subgoals the manager accepts. The worker's weights are never changed. This keeps the fix cheap and isolates the interface as the object of study.

Research Program

The first step is diagnostic. We train DIPPER in simulation on its four benchmarks: Maze, Pick and Place, Push, and Franka Kitchen. Following RGAT's protocol, we then deploy it to "surrogate-real" variants. These are the same environments with altered friction, added actuator latency, reduced torque limits and heavier objects.

For each failed episode we assign a cause. It is an interface failure if the proposed subgoal was unreachable by the real worker from that state, as checked by repeated execution. It is a worker failure if the subgoal was reachable in repeated trials but the worker failed on this attempt. If interface failures are a minority, the premise is wrong and the project should pivot to worker adaptation.

The second step builds the recalibrated critic, the active probing rule, and manager retraining under the new constraint. We study how the number of real subgoal trials affects success, and whether the fitted $\alpha,\beta$ alone recover most of the gain. If they do, the gap is mostly a shift in scale rather than one that depends on state.

The third step is a hardware check. We use the robosuite-to-Franka Panda transfer setting of MAPLE. MAPLE's parameterized primitives act as the worker, and its analytical affordance masks are replaced by the grounded reach probability. This tests whether the approach extends from DIPPER's learned worker to a primitive library whose feasibility was hand-specified.

Baselines

The naive reference is DIPPER trained in simulation and deployed zero-shot on the surrogate-real system with no correction. DIPPER's paper reports "up to 40% improvement in success rates" over SAGA, HAC and PIPER on its benchmarks, and HPO reports "up to 35%" over its baselines. No reported result was found on Lacuna for DIPPER under any reality gap, so we will measure that number ourselves.

The strongest competitor is simulator grounding followed by full retraining. We fit RGAT's action transformer $g_\phi(s_t,a_t)\to\hat a_t$ on real transitions, then retrain both worker and manager inside the grounded simulator. This is the obvious way to handle the gap, and it corrects dynamics everywhere rather than only at the interface. RGAT reports matching a policy trained directly on the "real" environment for flat control, but no reported numerical returns were found on Lacuna for RGAT. For fairness, it receives exactly the same number of real environment steps our method spends on probes, counting every step of every probe rollout. It also gets the same simulation compute and the same hyperparameter search budget.

A second competitor is DIPPER with a worker trained under broad domain randomization, meaning random variation of mass, friction, damping and latency during training. Its value function is then an average over dynamics.

A third competitor fine-tunes the worker with Soft Actor-Critic on the same real-step budget and recomputes $V^{\pi L}$ from the fine-tuned worker.

Two ablations isolate the key ingredients. The first replaces active boundary probing with uniformly sampled subgoal probes at equal budget. The second applies the grounded estimate only at execution time, rejecting and resampling infeasible subgoals, instead of using it to retrain the manager. We count as beating a baseline a statistically reliable gain in surrogate-real success at matched real-step budget, across at least three gap types and five seeds. We regard it as a clear win if our method reaches the grounded-retraining baseline's success with half the real steps.

Why not just…?

Why not just ground the simulator with RGAT and retrain everything? This is the real threat. With enough real data it may simply win, because it also fixes worker failures that our method ignores. Our bet is on sample efficiency. A binary reach outcome per subgoal, collected near the decision boundary, is a far lower-dimensional target than full transition correction. RGAT is also shown only on flat, single-tier control, so its behavior on long-horizon hierarchies is unknown. The matched-budget curves settle this. If grounded retraining dominates at every budget, the proposal's core claim is false. We will then report the result as evidence that interface miscalibration is not the bottleneck.

Why not just domain-randomize the worker so its value function is already robust? A randomized value function estimates reachability averaged over a distribution of physics. The actual robot is one point in that distribution, so the estimate can be too optimistic for some subgoals and too pessimistic for others. The file notes that broad randomization makes contact-rich policies overly conservative. Bridging the Sim-to-Real Gap for Athletic Loco-Manipulation finds that explicit actuator calibration beats randomization on hardware. We run randomization as a baseline and measure how well each method's predicted reach probability matches actual outcomes, to show which direction its errors go.

Why not just fine-tune the worker on real data, or retry when a subgoal fails? Fine-tuning changes the worker the manager was trained around. This reintroduces the non-stationarity that DIPPER's frozen, offline formulation was built to remove. It also spends real samples improving worker competence even when most failures come from the manager asking for impossible subgoals. Our diagnostic step measures that split. Retry-on-failure spends real time and risk on subgoals the robot cannot reach. The execution-time-filter ablation shows how much of the gain comes from retraining the manager rather than from rejecting subgoals at run time.

Proposed Experiments

The main experiment covers the four DIPPER benchmarks, each with four surrogate-real gap types and two gap magnitudes. We report surrogate-real success rate against the number of real environment steps. We also report the fraction of proposed subgoals the real worker cannot reach, and a calibration error: expected calibration error and Brier score of $\hat p$ against measured reach outcomes. The decisive checks are three. First, the diagnostic must show that interface failures are a substantial share of losses. Second, the method must beat zero-shot DIPPER and the domain-randomization baseline. Third, it must match or beat RGAT-grounded retraining at low real-data budgets.

The hardware check uses MAPLE's Stacking and Cleanup tasks on a physical Franka Panda. MAPLE reports "90% average success rate" in robosuite and "93%" and "83%" on hardware for these two tasks. We will introduce deliberate hardware perturbations: added payload, and a reduced-friction gripper pad. These should break its hand-written affordance masks. The test is whether a grounded reach estimate, fit from a few dozen real primitive calls, restores success to near the unperturbed hardware numbers where the analytical masks do not.