Subgoal-Level Credit for the Reality Gap

A robot policy trained in simulation often fails on a long task in the real world, and the failure usually has a single location: the gripper slips on one grasp, or a block slides farther than simulated on one push. Current transfer methods cannot find that location. They correct the simulator everywhere, matching every one-step transition, or they adapt one global parameter vector. We propose to use a two-level policy, in which a high-level planner picks intermediate targets ("subgoals") and a low-level controller reaches them, as a set of checkpoints. At the end of each subgoal segment we compare where the real (target-domain) robot ended up with where the simulator predicted it would end up from the same start state. We then ask the planner's own value estimate how much that discrepancy cost the task. The result is a per-segment account of blame for the reality gap. It tells us which skills to re-ground, and it tells us when the planner itself chose badly under the new physics.

The opportunity comes from the problem of assigning a delayed outcome to the step that caused it, known as temporal credit assignment. This is hard over hundreds of motor steps and with only a handful of real episodes. Over a dozen subgoal decisions it becomes tractable. A hierarchy shortens the horizon over which blame must be assigned. A simulator that can be reset to observed states supplies a counterfactual for each segment. Together they let a small real-data budget go where it matters.

Core Claim

When the sim-to-real gap is concentrated in a few parts of a long task, weighting the adaptation effort by per-subgoal "gap cost" gives higher target-domain success than uniform grounding when only 5 to 50 target episodes are available. It also separates two failure types that current methods conflate: controller-level execution error, where the subgoal was missed, and planner-level choice error, where the subgoal was reached but was the wrong subgoal under real physics. When the gap is global and uniform, for example a constant gravity change, we expect no advantage over uniform grounding. The experiments are designed to show both regimes.

Where This Sits

Three lines of work motivate the idea. On Credit Assignment in Hierarchical Reinforcement Learning proves that a one-step high-level update equals a flat multistep update with "skip connections" that jump over intermediate exploratory actions. It demonstrates this only in tabular grid mazes. We reuse the insight that the high level is the right place to assign credit, and apply it with deep value functions to a transfer problem. Hindsight Credit Assignment reweights actions by how likely they were given the eventual outcome, on small discrete diagnostics. Beyond Simple Sum of Delayed Rewards (CoDeTr) uses a transformer to break a delayed reward into step-level importance weights over 500-step horizons. Both are single-domain and flat. We move the attribution to subgoal segments, and we attribute the gap between two dynamics rather than the reward.

Two lines supply the grounding mechanism we redirect. Reinforced Grounded Action Transformation (RGAT) trains an action-correcting network with reinforcement learning, rewarding it for making each simulated transition match the recorded real one. Latent Adaptation of Foundation Policies (Found-adapt) infers a dynamics signature from a small batch of target transitions and adapts a frozen policy's skill latent in about 6 seconds. Both treat every transition as equally important and ignore downstream task outcome. We keep their machinery and change only which transitions they fit, and how strongly.

The hierarchy itself comes from DIPPER, described in Direct Preference Optimization for Primitive-Enabled Hierarchical RL and Hierarchical Preference Optimization. DIPPER trains the planner on a stationary set of subgoal comparisons, with a penalty from the controller's value for unreachable subgoals. That stationarity is also its limitation, and it is the tension this proposal targets. DIPPER's feasibility penalty is computed in simulation. Once physics shifts, it confidently endorses subgoals that are no longer reachable, or no longer useful. MetaWorld-X shows hierarchies help in high-DoF loco-manipulation, but it has no mechanism for diagnosing which routing choice caused a late failure.

One piece of prior work constrains the design. RADAR shows that visual shifts alone can collapse physical-robot success (to 0/32 under texture change). We deliberately scope this proposal to the physical dynamics gap, with state-based observations. A visual gap would contaminate the segment-end comparisons with perception errors that our attribution cannot separate.

Mechanism

The planner emits subgoal $g_k$ from state $s_k$ every $c$ steps, and the controller runs until the segment ends in target state $s_{k+1}$. Offline, we reset the simulator to $s_k$ and roll the same controller toward $g_k$, obtaining the simulated end state $\hat s_{k+1}$. The per-segment gap cost is

$$\Delta_k = V^H(\hat s_{k+1}, G) - V^H(s_{k+1}, G),$$

where $V^H$ is the planner's value function for final goal $G$. A large positive $\Delta_k$ means the reality gap in segment $k$ destroyed task value. A near-zero $\Delta_k$ with a large end-state discrepancy means the discrepancy did not matter.

A second quantity catches planner-level errors. When $\Delta_k \approx 0$ but the episode still fails, the fault is not in executing segment $k$. It lies in the planner's valuation of the subgoals it chose. We detect this with a small outcome-conditioned return decomposition over the $K$ segments of each target episode. This is CoDeTr's attention decomposition, applied to roughly 10 segment tokens instead of 500 steps.

Adaptation then splits. For the controller, we run RGAT or a Found-adapt latent update with each transition weighted by its segment's normalized $\Delta_k$. The RGAT reward becomes $r_t = -w_{k(t)} \lVert s^{\text{sim}}_{t+1} - s^{\text{tgt}}_{t+1}\rVert$, where $w_{k(t)} \propto \max(\Delta_{k(t)}, 0)$ for the segment containing step $t$. For the planner, we recompute DIPPER's feasibility penalty with target-grounded controller values, and we down-weight subgoals the segment decomposition blames.

We accept one known weakness: $V^H$ is trained in simulation and may misjudge states the target domain visits. The oracle comparison below measures how much this costs.

Research Program

We first build the hierarchy exactly as DIPPER does on its four tasks: 7-DoF Maze Navigation, Pick and Place, Push, and Franka Kitchen. Target domains follow the protocol RGAT and Found-adapt use, a modified simulator standing in for reality. There are two families. Global shifts reuse Found-adapt's gravity levels G1–G4 and friction levels F1–F6. Localized shifts are new: a low-friction patch on part of the table, a heavier object only after grasp, a stiffer microwave hinge in Kitchen. Because we inject the localized shifts, the true blamed segment is known. That makes attribution accuracy directly measurable, which is rarely possible in credit-assignment work.

The second stage studies the adaptation budget. We sweep 5, 10, 20, and 50 target episodes and report target success at each. The third stage, if resources permit, repeats Push on a physical arm with state estimation standing in for simulator reset. We treat it as a sanity check, not the main evidence, because the file contains no hardware results for any of the grounding baselines.

Baselines

The naive reference is direct transfer. The DIPPER hierarchy is trained in the nominal simulator and deployed unchanged. Beating it only shows the gap is real. A second reference is the same hierarchy trained with broad domain randomization over the perturbation ranges. This tests whether robustness at training time makes adaptation unnecessary. It uses equal simulator steps.

The strongest existing competitor is uniform grounding of the controller. We run RGAT and a Found-adapt-style latent adapter on all target transitions with equal weight, followed by DIPPER's planner unchanged. These are the methods built for exactly this problem, and our method differs from them only in weighting and planner correction. For the planner-level baseline, DIPPER itself reports "up to 40% improvements over state-of-the-art baselines" (HAC, SAGA, PIPER) on its suite in the nominal domain. Found-adapt reports consistent gains over direct transfer, GAT, and UGAT across gravity and friction shifts, but gives no exact success percentages. There is no reported result on Lacuna for any grounding method's target-domain success on DIPPER's tasks, so we produce these numbers ourselves.

Fairness rules apply to every method. All methods receive the same target episodes, the same simulator reset access, and the same hyperparameter search budget of 20 configurations tuned on a held-out perturbation. Each run uses 10 seeds.

Two ablations isolate the key ingredient. The first, distance-weighting, replaces $\Delta_k$ with DIPPER's Subgoal Distance Metric, meaning raw end-state error, so adaptation goes wherever the controller missed. This asks whether valuing the discrepancy matters beyond measuring it. The second, flat decomposition, applies CoDeTr over all low-level steps of the target episodes to produce transition weights, with no hierarchy. This asks whether the subgoal level matters.

An oracle weights grounding by the true perturbed segment. The gap between our method and the oracle bounds the cost of attribution errors.

We count as beating the baselines only the following outcome on localized shifts. Target success must exceed uniform RGAT/Found-adapt with non-overlapping 95% confidence intervals at 10 and 20 target episodes, and our method must beat distance-weighting on the Push and Kitchen shifts, where reached-but-useless subgoals are expected.

Why not just…?

Why not just ground every transition uniformly with RGAT or Found-adapt? For global shifts this may be optimal, and we predict parity there. With a small target budget and a localized shift, most transitions carry no information about the gap. Uniform fitting then spends capacity matching irrelevant regions and can distort dynamics that were already correct. The global-versus-localized comparison settles it. If uniform grounding matches our method on localized shifts at 10 episodes, the core claim is wrong.

Why not just weight by how far the controller missed each subgoal? This is the most serious threat, because miss distance is cheap and correlates with gap cost. It fails in two predictable cases. A large miss can be harmless, for example when the arm overshoots in free space and the next subgoal absorbs it. A subgoal can be reached exactly and still doom the task, for example a push start position that works under simulated friction but not under real friction. Distance-weighting is run as a named ablation. If it ties our method across all tasks, we will report that the value-based weighting is unnecessary and that the contribution reduces to subgoal-segmented grounding.

Why not just apply return decomposition to the flat trajectory? CoDeTr handles 500-step delays with ample data. Here the budget is tens of episodes, and the skip-connection argument of On Credit Assignment in Hierarchical Reinforcement Learning predicts that a roughly 10-token decomposition is far easier to fit than a 500-token one. That argument is tabular, however, and the flat-decomposition ablation tests it in continuous control.

Why not just train with broader domain randomization? Randomization is widely reported to produce overly cautious policies on contact-rich tasks. The randomization baseline measures that trade-off directly. It can win if the shifts lie inside the randomized range, so we also evaluate it on shifts just outside that range.

Decisive Checks

The first check is attribution accuracy. On localized shifts, the highest-$\Delta_k$ segment should coincide with the injected perturbation (segment-level AUROC well above distance-weighting). If it does not, the downstream success gains, if any, are not coming from the proposed mechanism.

The second check is the budget curve. Our advantage over uniform grounding should be largest at small budgets and shrink as target data grows. That shape follows from weighting being a data-efficiency device, not a capacity one.

The third check is the planner/controller split. On the reached-but-wrong-subgoal shifts, correcting only the controller should recover little, and adding the planner correction should recover most of the gap to the oracle.

Together, these three checks either establish segment-level blame as a practical tool for spending scarce real-world data, or show that simple miss distance already does the job.