Subgoals That Know the Physics: Online Feasibility Estimation for Hierarchical Sim-to-Real Manipulation

Robots that solve long, multi-step tasks are often built in two layers. A slow "planner" picks intermediate targets, called subgoals ("put the gripper 3 cm above the block"). A fast "controller" moves the motors to reach each target. These systems are usually trained in simulation, and the simulator's physics is varied at random during training (masses, friction, motor strength, delays) so the policy survives the mismatch with the real robot. This practice is called domain randomization. Randomization and hierarchy interact in a way nobody has dealt with directly. Once physics varies, whether a subgoal is reachable stops being a fixed fact about the controller and becomes a fact about the controller in this particular world. A subgoal that is easy on low friction may be out of reach under heavy damping or a weak motor. Today's planners either ignore this, or, if they model reachability at all, assume one fixed physics.

We propose that the planner should infer the current physics from the controller's recent successes and failures at reaching subgoals, and then propose only subgoals that are feasible under that inferred physics. The record of which recent subgoals were reached and which were missed is a cheap probe of the hidden dynamics, and it is already available inside any hierarchical agent. We train a feasibility model conditioned on this history across a randomized distribution of physics. At deployment, including on a real robot, the planner re-estimates its reachable region within a handful of subgoal attempts. This avoids the overconfident subgoals that come from assuming nominal physics. It also avoids the paralysis that comes from assuming the worst case over all randomized physics.

Core Claim

Prior work checks whether a subgoal is feasible only under fixed physics. Prior work that does randomize physics across levels never models subgoal feasibility. DIPPER (Direct Preference Optimization for Primitive-Enabled Hierarchical RL) and its companion HPO (Hierarchical Preference Optimization) penalize subgoals that the controller's value function scores as unreachable, but they train and test in unrandomized simulation. Strict Subgoal Execution (SSE) records strict reach-or-fail outcomes to map a reachability frontier, again under one physics. Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real supplies the key constraint. Heavy randomization on a single flat policy produced 0% real-world success. Putting heavy randomization on the controller and light randomization on the planner produced 73.3–90%. But its planner issues 2D waypoints and never asks whether they are reachable.

Our claim is that, under randomized physics, subgoal feasibility should be treated as a quantity conditioned on the current dynamics and inferred online. Two simpler choices fail in opposite directions. Averaging feasibility over the randomized physics is overconfident in hard worlds. Taking the worst case over them recreates the conservatism that modular randomization was introduced to fix. The claim is falsifiable: if a planner given the full history without an explicit feasibility model, or a worst-case planner, matches our method on real-robot success and completion time, the claim fails.

Research Program

The low level is deliberately conventional. Following MAPLE, it is a library of parameterized behavior primitives (reach, grasp, push, release, atomic) that runs on an operational-space controller. The subgoal space is specified by hand as gripper and object Cartesian targets. This is the "manual decomposition" direction used as an interface, not as the object of study. We train the controller with heavy dynamics randomization over link masses, joint friction, actuator gain, object mass and friction, and control latency. This follows the dynamics-randomization recipes of Sim-to-Real Transfer of Robotic Control with Dynamics Randomization and Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. MAPLE itself transferred zero-shot with no dynamics randomization, so adding it is itself a tested change.

The new component is a feasibility model, written $F_\phi(g \mid s, h)$. It predicts the probability that the controller reaches subgoal $g$ from state $s$ within its step budget. Here $h$ is the history of the last $m$ subgoal attempts, each recorded as the tuple (state, subgoal, reached state, strict success flag). Labels come from SSE-style strict termination rather than hindsight relabeling, because relabeling a miss as an intended endpoint destroys exactly the signal we need. During training, the critic has privileged access to the sampled physics parameters, as in Asymmetric Actor Critic. We add an auxiliary loss that asks a history encoder to predict those parameters, so that $h$ carries information about the dynamics. Only $h$ is available at deployment. The planner, a soft actor-critic over primitive choice and parameters, then solves

$$\max_{\pi_H}\; \mathbb{E}\big[R\big] \quad \text{subject to} \quad F_\phi(g_t \mid s_t, h_t) \ge \tau \ \text{for every issued subgoal } g_t,$$

where $R$ is the sparse task return and $\tau$ is a feasibility threshold. The constraint is handled with a Lagrange multiplier, the same relaxation DIPPER applies to its value-function penalty. Our method differs from DIPPER in what the constraint depends on, which is $h$, and in how the constraint is labeled, which is by strict outcomes rather than the controller's value.

Following the modular principle, the planner sees only light noise and never sees the raw dynamics parameters. The analytical part of the program measures calibration. For a held-out physics setting, we compare predicted feasibility with the empirical reach rate as a function of how many attempts are in the history. This gives a concrete adaptation curve: how many subgoal attempts are needed before the frontier is right. If this curve does not flatten within roughly five to ten attempts, the method is too slow for real episodes, and we would report that.

Baselines

The naive reference point is a flat soft actor-critic policy over atomic actions, trained with the same dynamics randomization. MAPLE reports 19% average success for atomic-only SAC across eight robosuite tasks without randomization. The hierarchical sim-to-real paper reports 0% real-world success for flat randomized policies. We expect flat randomized SAC to fall well below the hierarchical methods. We run it only to confirm that the regime is one where hierarchy matters.

The strongest existing system is MAPLE retrained under our dynamics randomization, with its own affordance checks and no feasibility model. MAPLE reports 90% average simulated success over eight robosuite tasks, and 93% on Stacking and 83% on Cleanup zero-shot on a real Franka Panda. We take those figures as the bar on nominal physics. Beating MAPLE means higher success on the held-out hard-physics band in simulation, and no loss on nominal physics, at an equal number of environment steps and an equal hyperparameter search budget of the same number of seeds and sweeps.

The strongest feasibility-aware competitor is DIPPER-style regularization by the controller's value function, trained under randomization but not conditioned on history. DIPPER reports up to 40% improvement in success over SAGA, HAC and PIPER on its benchmark suite, in unrandomized simulation. No reported result found on Lacuna for DIPPER or SSE under dynamics randomization, so we produce these numbers ourselves. We use the same primitive library and controller so that only the planner's feasibility signal differs. We also add an SSE-style baseline that applies strict reach-or-fail labeling with a single unconditioned feasibility model.

Two ablations isolate the key ingredient. In the first, the worst-case ablation, $F$ is replaced by its lowest quantile across randomized physics. This tests the conservatism hypothesis. In the second, the implicit-history ablation, the planner is a recurrent policy that receives $h$ but has no explicit feasibility constraint. This tests whether the explicit model matters at all. Our method counts as a success only if it beats the worst-case ablation on completion time and the implicit-history ablation on success under held-out physics.

Why not just…?

Why not just train the planner under the same randomization and let it average over feasibility? That is our DIPPER-under-randomization baseline. Averaging gives a single reachable region that is too large in hard worlds and too small in easy ones. The hierarchical sim-to-real result shows that the planner's exposure to dynamics matters a great deal. The calibration experiment, which measures predicted versus actual reach rate per physics band, directly shows whether averaging is miscalibrated at the extremes. If it is not miscalibrated, our method has no reason to exist.

Why not just be robust: constrain subgoals to be feasible under the worst-case physics? This is the most natural "risk-aware" version of the gap. It is a plausible threat on tasks with loose geometry, where small, cautious subgoals cost little. We expect it to lose on completion time and on tasks that need dynamic motions such as pushing to a far target or cleanup. That expectation follows from the documented collapse of monolithic randomization into over-conservative behavior. The worst-case ablation reports both success and steps-to-completion so that the trade-off is visible rather than hidden.

Why not just give a recurrent planner the history and skip the explicit feasibility model? This is the most serious alternative. A recurrent planner trained by RL could in principle learn the same adaptation implicitly. Our bet is that sparse task reward is too weak a signal for the planner to learn calibrated reachability. Strict success flags give dense, direct supervision. We do not treat this as settled. The implicit-history ablation receives the same inputs and compute, and if it matches us we will report that the explicit feasibility model is unnecessary.

Why not just do system identification first and plan with known parameters? Estimating friction or latency on a real arm takes a separate procedure for each deployment. It also identifies parameters that may not matter for reachability. We include a variant that conditions on the true parameters as an upper bound in simulation, to measure how much of that oracle the history-based estimate recovers.

Proposed Experiments

In simulation, we use robosuite (robosuite) with MAPLE's eight tasks. We add a dynamics-randomization wrapper and split physics into a training band and a held-out "hard" band with higher friction and damping, lower actuator gain, and added latency. We repeat the comparison on the DIPPER suite, which covers Pick and Place, Push and Franka Kitchen, using its subgoal-distance diagnostic: the distance between the proposed subgoal and the state actually reached. To check that the idea is not specific to manipulation primitives, we also run SSE's AntDoubleKeyChest under randomized leg actuation. SSE reports nearly 100% success there with fixed physics, while HIRO and HIGL score 0%. The metrics are task success, steps to completion, feasibility calibration error per physics band, the attempts-to-calibration curve, and the subgoal distance metric.

The decisive simulation check has two parts. On the held-out hard band, our method must exceed both MAPLE-with-randomization and DIPPER-with-randomization in success. It must also beat the worst-case ablation in completion time while staying within noise of it in success. On real hardware we follow MAPLE's protocol on a Franka Panda for Stacking and Cleanup, and add physical perturbations: a weighted block, a high-friction mat, and an artificially lowered controller gain. Our method must hold near MAPLE's reported 93% and 83% on unperturbed trials and degrade less than MAPLE-with-randomization on perturbed ones, over at least 30 trials per condition. A null result in which the recurrent-history or averaged baseline matches us on perturbed hardware would falsify the core claim, and we would report it as such.