Executor-Grounded Online Subgoal Selection for Hierarchical Robot Control

Many capable robot systems now split control into two parts. A slow planner reads an instruction and proposes an intermediate goal, such as an image of what the scene should look like a few seconds from now. A fast controller then drives the arm toward that goal. The weak point is the handoff between them. The planner has no idea what the controller can physically reach, so it sometimes proposes goals that look plausible but cannot be executed. The best current fix trains a filter offline and then freezes it. We propose that the robot instead learn which proposed subgoals its own controller can actually reach, and learn this online during deployment. It gets the needed feedback for free: after each attempt, it compares where it ended up with where it was told to go. No hand-written reward is required.

We cast choosing one of several candidate subgoals as a contextual bandit. A contextual bandit is a repeated choice problem: the learner sees some side information (the context), picks an option, and sees a reward only for the option it picked. The reward signal comes from the controller itself. The learning algorithm is chosen so that its guarantees do not assume the situations it faces are drawn independently from a fixed distribution. That assumption fails for a deployed robot, because the scenes it meets depend on its own earlier actions and on shifts in its environment. In two sentences: generate several candidate subgoals, let an online learner pick among them using a feasibility score trained only on whether the frozen controller actually reached past picks, and use a bandit algorithm whose regret stays bounded even when the scenes are chosen adversarially. This yields a subgoal interface that adapts to the controller's real limits, needs no manual reward, and degrades gracefully under worst-case scene shift.

Core Claim

SuSIE generates subgoal images with an image-editing diffusion model and tracks them with a goal-conditioned diffusion policy. Its authors note that the planner is blind to the controller's kinematic envelope. GHIL-Glue attacks this with an action-free classifier, trained offline, that rejects "off-task" candidates, plus mismatched image augmentations for the observation and the subgoal. The classifier is fixed at deployment and judges task progress in general, not whether this particular controller in this particular scene can get there. Our claim is that the missing piece is a feasibility estimate that is grounded in the controller and updated online.

Our second claim concerns where the theory should live. The adversarial-bandit guarantees in the file require realizability, meaning the true reward function lies in the learner's model class. As the research file points out, a diffusion generator plainly violates this, since it hallucinates. We therefore never place the guarantee on the generator. We place it on the scoring function that maps (current observation, candidate subgoal) to the probability that the frozen controller reaches the subgoal. That function is well defined and low dimensional in its output, and a small head on frozen visual features can plausibly represent it.

Third, and this is where manual reward engineering enters as a constraint rather than a topic, a reachability-only reward is trivially hackable. The learner would happily pick "subgoals" that are nearly the current frame. We avoid this by splitting the decision in two:

  • Progress stays with GHIL-Glue's frozen classifier and acts as a gate: only candidates above a progress threshold are eligible.
  • Feasibility is the only quantity learned online, among the gated candidates.

The online learner never optimizes a signal it could inflate by making no progress.

Research Program

At each high-level step $t$, the planner samples $K$ candidate subgoal images. The GHIL-Glue classifier removes candidates below its progress threshold. Each surviving candidate $a$ is described by features $x_{t,a}$ taken from a frozen visual encoder applied to the pair (current frame, candidate). An online regressor $\hat f_t$ predicts the reachability reward for each candidate. The selector then draws a candidate by inverse-gap weighting, the exploration rule behind the regression-reduction bandits of Contextual Bandits with Online Neural Regression:

$$p_t(a) = \frac{1}{|A_t| + \gamma\,(\hat f_t(x_{t,\hat a}) - \hat f_t(x_{t,a}))}\quad (a \neq \hat a),$$

Here $A_t$ is the surviving candidate set, $\hat a$ is the candidate with the highest prediction, which receives the remaining probability mass, and $\gamma$ is a learning-rate-like parameter that controls how sharply exploration focuses on near-best candidates.

After the frozen controller runs its fixed window of steps, we compute the reward. It is an asymmetric reachability score between the frame actually reached and the chosen subgoal, of the kind Mezghani et al. learn with a directional reachability network from unlabeled trajectories. We train it on the same robot data the controller used. Asymmetry matters here because "can I get from here to there" is not symmetric once grasps and drops are involved. The regressor is updated on that single observed reward.

Rather than following that paper's exploration-free greedy analysis, we adopt this reduction for two reasons:

  • Its regret bound, of order $\tilde{O}(\sqrt{KT})$ for squared loss, holds without independent, identically distributed contexts. The authors also report the lowest cumulative regret among neural bandits under an adversarial context-shift protocol on OpenML tasks.
  • Qi et al. show that optimism-based neural bandits can suffer linear regret when arms have near-identical representations. Candidates sampled from one diffusion model given one prompt are exactly such near-duplicate arms.

The research program has three parts. The first is building the selector on top of released SuSIE and GHIL-Glue components, with the controller and generator frozen, so the only thing that learns during deployment is a small scoring head.

The second is a regret analysis stating what the reduction does and does not buy here. The comparator is the best feasibility scorer in the model class in hindsight, not the best possible plan. Two things the guarantee does not cover must be stated honestly:

  • Reward noise from the learned reachability model. We treat this as label contamination, and we test sensitivity to it. The spectral robust-regression tools of Online and Distribution-Free Robustness indicate what a principled fix would cost. The file notes their SoS variant is too heavy for real-time loops, so we would test only the cheaper SCRAM variant if contamination proves damaging.
  • The fixed window between subgoals. PlanDQ shares this rigidity, and we inherit it rather than solve it.

The third is an adversarial evaluation protocol for hierarchical manipulation, which the file shows does not currently exist.

Baselines

The naive reference is SuSIE with a single sampled subgoal and no filtering. Its reported CALVIN zero-shot result is 2.69 instructions completed in a row, against 1.43 for AugLC (SuSIE). We rerun it on the same seeds as every other arm rather than relying on the reported figure.

The strongest existing method is GHIL-Glue: SuSIE plus the offline subgoal classifier with rejection sampling plus mismatched augmentations. It reports a 25% improvement in average successful sequence length on CALVIN and a 23% absolute success gain over SuSIE on the Bridge V2 real platform. No absolute CALVIN sequence length for GHIL-Glue was found on Lacuna, so we reproduce it. It is the right main competitor because it attacks the same failure, off-task and infeasible subgoals, using the same generator and controller. We run it with the same $K$ candidates per step, the same frozen controller, and the same evaluation sequences.

To beat it, the proposal must reach a higher average sequence length on standard CALVIN held-out environment D. The expected margin there is small. The meaningful claim is a clearly larger margin, with non-overlapping seed confidence intervals, on the adversarial CALVIN variant described below.

Two ablations isolate the key ingredients:

  • Greedy online selector. The same online regressor and reward, but the selector always picks the top prediction. This separates the value of controller-grounded online learning from the value of principled exploration.
  • Offline-fitted feasibility head. The same reward model, but the head is fit once on held-out rollouts and then frozen. This tests whether being online matters at all, or whether a better offline feasibility score is enough.

A NeuralUCB-style selector gives the optimistic-exploration comparison. For long-horizon context, we report the L-CALVIN reference of 4.75 tasks for Long-VLA, a single-model system. It is not directly comparable to a subgoal selector, but it shows how far hierarchical selection sits from the monolithic frontier.

All online methods share the same encoder, head architecture, reward model, and a tuning budget of equal size over $\gamma$ or its analogue. Tuning uses CALVIN training environments only.

Why not just…?

Why not just sample more candidates and keep GHIL-Glue's frozen classifier? This is the most direct threat. If infeasibility were mostly a matter of sampling luck, best-of-$N$ would recover most of the gain. But the frozen classifier scores progress, not reachability by this controller. When the scene shifts, such as new lighting, displaced objects, or distractors, a larger $N$ gives it more ways to select a subgoal that looks good but that the controller cannot reach. We test this by sweeping $N$ for GHIL-Glue until its compute per step matches or exceeds ours. If it closes the gap under the adversarial protocol, the online component is not justified, and we would report that.

Why not just update the classifier greedily online, with no exploration machinery? This is a real possibility. Sivakumar et al. prove that greedy least squares suffices when contexts are smoothed by random noise, and ordinary scene variation may be random enough. We expect greedy to match inverse-gap weighting on standard CALVIN and to fall behind only when the adversarial schedule concentrates on its blind spots. If greedy matches everywhere, the contribution shrinks to the controller-grounded reward and the progress/feasibility split, and we would present it that way.

Why not use optimism bonuses like NeuralUCB? Candidates from one diffusion prompt are highly correlated, which breaks the arm-separateness assumption. Qi et al. show that this failure drives optimistic neural bandits to linear worst-case regret. The NeuralUCB baseline tests whether this occurs in practice.

Why not just make the controller more robust, or use a large end-to-end model? Mismatched augmentation is already part of GHIL-Glue, so it is included in our strongest baseline. Our method is complementary to a better controller and could be stacked on one. The end-to-end route trades away the latency advantage that motivates splitting control, as argued by HiRT. It is also reported to lose to SuSIE on precision tasks against RT-2-X, though no numeric value for that comparison was found on Lacuna.

Proposed Experiments

Standard evaluation uses CALVIN held-out environment D with the usual five-instruction chains, L-CALVIN ten-step chains, and a small real Bridge V2 evaluation on four tasks mirroring GHIL-Glue's protocol.

The adversarial variant of CALVIN follows the OpenML shift protocol from Contextual Bandits with Online Neural Regression. Every fixed block of episodes, the scene sampler raises the probability of the configuration family where the current system fails most often. The families cover lighting, table texture, distractor objects, and initial object placement. The adversary thus adapts to the learner, which is the regime the guarantees are meant to cover.

The metrics are:

  • Average consecutive instructions completed.
  • Cumulative feasibility regret against a simulator oracle. The oracle uses privileged end-effector and object poses, only for evaluation, to judge which gated candidate was actually reachable.
  • Agreement between the learned reachability reward and that oracle.
  • A reward-hacking rate: the fraction of selected subgoals with near-zero ground-truth progress.

The decisive checks are four:

  1. Under the adversarial schedule, the online selector must beat GHIL-Glue at matched $N$, with confidence intervals over seeds that do not overlap.
  2. Feasibility regret must grow sublinearly with the number of high-level steps, while the frozen classifier's regret grows linearly.
  3. The gap between inverse-gap weighting and greedy must appear under the adversarial schedule and be small in the standard setting. That pattern would confirm that exploration earns its keep only where theory says it should.
  4. Removing the progress gate must raise the hacking rate sharply. This would show that confining online learning to feasibility is what keeps a controller-grounded reward safe to optimize.

A failure of the first check under matched compute would falsify the proposal's central claim.