Grounding-Gated Distillation for Long-Horizon Robot Policies

Large vision-language-action models (robot controllers that read a camera image and a text instruction and output motor commands) are too slow for many real robots. The natural fix is distillation: train a small, fast student policy to imitate the large teacher. For long, multi-stage tasks this runs into a specific problem. Small errors compound over a task, so the student soon visits states that never appear in the human demonstrations. The teacher's advice in those states is the most valuable signal available, and it is also the least trustworthy. Existing methods that decide how much to trust the teacher on each sample all compare the teacher's error and the student's error against a ground-truth label. In off-demonstration states no ground-truth action exists.

We propose to weight the teacher's supervision at each timestep by a label-free test of whether the teacher is actually grounded in that state. The test asks whether the teacher's action changes when the task-relevant part of the image is masked, and whether it changes when the instruction is swapped. A teacher that ignores one modality can still be confident, but it is acting from habit. Its advice in that state should count for little. A teacher that responds to both the image and the instruction should be trusted in proportion to how much the student disagrees with it. The claim is that this cross-modal grounding check, and not teacher confidence or loss ratios, is the right gate for on-policy distillation over long horizons. The resulting student should lose less success per added task stage than students distilled with uniform or confidence-based weights.

Why this gap exists

Adaptive distillation weighting is well developed for classification, and it assumes labels everywhere. Rethinking Soft Labels for Knowledge Distillation shows that teacher soft labels act as a per-sample regularizer. Its Weighted Soft Labels scale the distillation loss by the ratio of student to teacher cross-entropy against the true label. Sample-level Adaptive Knowledge Distillation for Action Recognition (SAKD) measures per-sample difficulty by perturbing video frames and falls back to ground-truth labels on hard samples. StructVPR++ and Relational Subsets Knowledge Distillation up-weight samples with large teacher–student gaps, and Hybrid Forward down-weights transfer when the teacher is misaligned with the task. We reuse the shape of these weighting functions. All of them, however, are evaluated on static datasets where a label is available to arbitrate. That is exactly what disappears once a policy drifts.

Work on long-horizon tasks motivates why drifted states matter. ClevrSkills finds that success falls sharply with horizon length even after fine-tuning on multi-step data, and that policies loop or fail to recover from small deviations. RLBench and ALFRED both locate failures at sub-goal transitions. BAKU is the compact student we have in mind: a roughly 10M-parameter multi-task transformer at 10 Hz. It is trained with uniform behavioral cloning, so it has no mechanism to focus training on drift-prone states.

Work on the cross-modal gap supplies the reason a teacher can be confidently wrong. Tuning LayerNorm in Attention treats visual tokens entering a text model as a distribution shift that decoders handle poorly. More broadly, global projections lead decoders to underuse non-text inputs. VIMA responds by building object-centric tokens, and AMLRIS prunes hard-to-align pixels during grounding. Both change the representation. We instead use misalignment as a measurable per-state signal that decides how much the teacher's advice counts.

Three papers are closest. BOLT distills a 13B multimodal model into a 2B student for robot question answering and routes compute adaptively. It works on multiple-choice answers, it adapts only at test time, and it has no closed loop. Any-OPD does on-policy distillation, with the teacher correcting the student's own rollouts, across mismatched representation spaces. Its domain is generative flow-matching models, and it has no per-state reliability weighting and no physical horizon. Action-Sketcher and Align-Then-stEer bridge language-to-action and cross-embodiment gaps for long-horizon manipulation, but neither compresses the model. Our contribution is the combination none of them has: on-policy distillation into a compact continuous-control student, with per-timestep weights computed without labels from the teacher's cross-modal grounding.

Core Claim

Let $s$ be the observation, $l$ the instruction, $\pi_T$ the teacher's action distribution and $\pi_S$ the student's. Let $\tilde s$ be the observation with the task-relevant region masked, and $\tilde l$ a different instruction valid in the same scene. The task-relevant region is the teacher's own attended region or, in simulation, the target object's mask. We define a grounding score

$$g(s,l) = \min\Big(D\big(\pi_T(\cdot\mid s,l),\,\pi_T(\cdot\mid \tilde s,l)\big),\; D\big(\pi_T(\cdot\mid s,l),\,\pi_T(\cdot\mid s,\tilde l)\big)\Big),$$

where $D$ is a distance between action distributions. We use mean squared distance between predicted action chunks for deterministic heads and sampled-action energy distance for diffusion heads. The score is normalized to $[0,1]$ by its running percentile. A teacher earns a high $g$ only if it reacts to both modalities. The per-timestep distillation weight then mirrors the form of Weighted Soft Labels, but replaces the label-based loss ratio with label-free disagreement:

$$w(s,l) = g(s,l)\,\Big(1 - \exp\big(-D(\pi_S,\pi_T)/\tau\big)\Big),$$

with temperature $\tau$. The student minimizes behavioral cloning loss on demonstration states plus $w$-weighted distillation loss on states drawn from its own rollouts.

The falsifiable prediction has three parts. First, $g$ should predict whether the teacher can actually finish the task from a given drifted state better than teacher entropy or teacher–student disagreement do. Second, students trained with $w$ should show a flatter decline in success as the number of task stages grows. Third, the gain should concentrate in stages that occur after the student's first deviation from demonstration states.

Research Program

The work has three parts.

The first part is a reliability study. It decides whether the idea is worth pursuing, and it needs no new training method. In simulation we can measure teacher reliability directly. From each student-visited state we hand control to the teacher and record whether it completes the task. We call this the takeover success rate. We will collect states from rollouts of a behavior-cloned student, label each with its takeover success, and compare how well several signals rank those states. The signals are $g$, its vision-only and language-only halves, teacher action entropy (the signal behind EA-KD), teacher ensemble variance, and SAKD-style temporal perturbation sensitivity. If $g$ does not beat entropy here, the method reduces to entropy-weighted distillation and we stop.

The second part is the distillation method itself. The loop alternates three steps: roll out the student, have the teacher label the visited states with its actions plus the two counterfactual queries, and train the student on demonstrations plus weighted teacher labels. Each labeled state costs three teacher forward passes. All of them happen offline, so the deployed student has no added inference cost. We will also test a stage-aware variant. It resets the running percentile for $g$ at detected sub-goal boundaries, so that hard transitions are not judged against easy mid-stage states.

The third part is analysis. We will plot success against stage index and against horizon length. We will localize failures to transitions versus within-stage execution. We will also check whether the distribution of $g$ shifts across stages, which would show that teachers lose grounding late in long tasks.

Proposed Experiments

The primary benchmark is LIBERO's long-horizon suite, with LIBERO-90 for multi-task breadth, both from LIBERO. RoboTwin 2.0, used by Action-Sketcher, serves as a bimanual check. VIMA-BENCH's L1–L4 generalization levels test whether grounding-gating helps on novel objects and tasks, where cross-modal failure should be most common.

The teacher is a large vision-language-action model fine-tuned on each suite. We use π0 or OpenVLA, both adapted in Align-Then-stEer. The student is the BAKU architecture with a multilayer-perceptron action-chunk head, chosen for its 10 Hz budget. Metrics are task success rate, success rate per stage reached, takeover success rate for the reliability study, and student control frequency. Every method gets the same number of teacher-labeled states and student gradient steps, with three seeds per method.

Baselines

The naive reference point is the BAKU student trained by plain behavioral cloning on the same demonstrations, with no teacher. The relevant reported figure is BAKU's 90% success on LIBERO-90 (BAKU). We will retrain it ourselves under our splits. We found no reported result on Lacuna for BAKU or a distilled compact student on the LIBERO long-horizon suite specifically, so that number will be established in this work. Beating this reference means that the teacher's on-policy labels add anything at all.

The strongest competitor on this question is uniform on-policy distillation. In this DAgger-style scheme the teacher labels every student-visited state and all labels are weighted equally, as in the on-policy setup of Any-OPD transplanted to control. It directly attacks compounding error. It uses the same teacher queries and the same student, and it is tuned with the same budget over learning rate and the mixing ratio of demonstration to on-policy data. Our method beats it if it improves long-horizon success by more than the seed-to-seed spread, with the margin growing with stage count.

Two adaptive-weighting baselines isolate the gating signal. One is entropy weighting in the spirit of EA-KD. The other is Weighted Soft Labels from Rethinking Soft Labels, applied only on demonstration states where labels exist and paired with uniform weights on on-policy states. Both get the same tuning budget over their temperature. We also include the teacher itself, as an upper bound on success and a reference point for latency.

The key ablations are $w$ with $g$ fixed to 1 (disagreement only), $g$ with vision-only or language-only counterfactuals, and an oracle weight equal to measured takeover success. The oracle tells us how much of the available gain $g$ captures.

Why not just…?

Why not just run on-policy distillation with uniform weights? This is the real threat. If the teacher is reliable almost everywhere students drift, gating adds nothing. The reliability study settles the question before any method claim is made. If takeover success is uniformly high on drifted states, uniform distillation is the answer and we will report that. The proposal is only worth pursuing if teachers frequently fail from student-induced states. The compounding failures documented in ClevrSkills, and the 25% ceiling on multi-stage tool use in Align-Then-stEer, suggest they do.

Why not just weight by teacher confidence or the student/teacher loss ratio? Loss-ratio methods such as Weighted Soft Labels and SAKD need a ground-truth action, which off-demonstration states lack. Entropy needs no label, but a model that ignores its visual input can be confident and wrong. That is the failure mode the visual-token work in Tuning LayerNorm in Attention points toward. Whether entropy is nonetheless as good a predictor as $g$ is an empirical question. The ranking comparison against takeover success answers it directly, and the entropy-weighted baseline answers it end to end.

Why not just skip distillation and deploy the big teacher with budget-aware routing, as BOLT does? BOLT routes discrete question answering, where one expensive query per decision is tolerable. Closed-loop control at 10–50 Hz needs an answer every cycle. The iterative heads in large action models already strain real-time control, as noted for Align-Then-stEer. We will report teacher latency alongside student latency so the trade-off is explicit.

Why not just up-weight sub-goal transitions with a fixed schedule? Transitions are where failures cluster, so this heuristic is cheap and plausible. We will include it as a baseline, with transitions detected from demonstration annotations. It cannot tell a transition where the teacher is grounded from one where it is guessing. If it matches our method, the effect is about timing rather than grounding, and we will say so.

Decisive Checks

The first decisive check is the reliability study: $g$ must rank drifted states by takeover success better than entropy, at a gap larger than the variation across seeds. The second is the horizon curve. On the LIBERO long-horizon suite, the grounding-gated student must lose less success per added stage than uniform on-policy distillation. The difference should appear mainly after the first deviation from demonstration states, while the two methods stay close on early stages. The third is the oracle gap: grounding-gating should recover a substantial fraction of the improvement that oracle takeover weights provide. If the first check fails, the project reduces to a careful negative result about teacher reliability under drift, which is still useful. If the first passes and the second fails, the grounding signal is real but distillation cannot exploit it at this student size. We would then test a larger student before drawing conclusions.