Fast Eyes, Slow Thoughts: Two-Rate Perception for Reactive Vision-Language-Action Policies

Vision-language-action models are robot controllers built on large pretrained vision-language networks: they read camera images and a typed instruction and output motor commands. They have become fast at the action end. Recent work compresses or parallelizes action generation until the decoder is no longer the slow part. The cost has moved upstream. The authors of FASTer cut action decoding to 112ms on LIBERO, then reported that the large image encoder now takes most of every control cycle. Running a billion-parameter vision stack on every frame is what keeps these policies off onboard robot hardware. It also slows their reaction when the world changes mid-motion.

We propose to stop running the heavy encoder at control rate. A slow pathway runs the full vision-language model every few hundred milliseconds and caches its semantic output: what the task is and where the relevant objects are. A fast pathway, a small image network, runs on every frame. It is trained to predict what the heavy encoder would say about the current frame, given the cached slow output. A one-pass parallel action decoder reads both pathways. The fast network is trained only to imitate the teacher's features, so it needs no action labels. It can therefore be trained on abundant action-free robot video and simulator renders, while the scarce demonstrations are spent only on the small action decoder.

Core Claim

For a vision-language-action policy, the latency that matters is the time from a change in the scene to a change in the robot's motion. That latency is set by the vision encoder, not the action decoder. Our claim is that a fast student encoder can preserve the success rate of a full-rate policy on static benchmarks and beat it on dynamic ones, while cutting reaction latency several-fold. The student is distilled from the teacher's features on action-free frames and conditioned on stale semantic context from a slow teacher pass.

The second half of the claim carries the data-scarcity argument. A small encoder trained this way should clearly beat the same small encoder trained end-to-end on the demonstrations alone. The demonstrations are too few to teach a compact network good visual features. Teacher features on unlabeled frames are not.

Relation to Prior Work

Three lines of work make action generation fast, and none of them touches perception cost. FAST compresses action chunks with a frequency transform and byte-pair encoding, but still decodes tokens one by one: 750ms per one-second chunk, against 100ms for diffusion. FASTer learns a quantized action tokenizer and decodes blocks of tokens per step. Discrete Diffusion VLA predicts all action tokens in parallel and refines them over 12 masked passes, reaching 14.53 Hz. One-Step Diffusion Policy distills a diffusion policy into a single-step generator. All four re-run the full visual stack whenever a new chunk is produced. Our proposal reuses their action representations and leaves the decoder question mostly settled. The new part is that the image encoder does not run at full size on every step.

Two papers supply the perception method. Multi-Modality Co-Learning uses heavy RGB and vision-language teachers only during training and discards them at inference, leaving a light skeleton network. Modality-specific Distillation shows that distilling per-modality teacher outputs beats monolithic distillation. On Hateful-Memes, its AUC is 68.30% versus 66.53% for plain distillation and 65.54% for a small model. Co-Learning works on classification and assumes a free cheap modality (skeletons). Robots have no such modality, so we keep a cheap camera stream and let the teacher stay in the loop at a low rate rather than vanish.

The decoder mechanism comes from speech recognition. Paraformer trains a one-pass parallel decoder with a "glancing" sampler, which feeds the decoder some true tokens during training in proportion to its current errors. This teaches dependencies between tokens that a single parallel pass otherwise ignores. With it, Paraformer matched autoregressive accuracy (7.71% vs 7.55% CER) at a tenfold speedup. LASO showed a 50x latency reduction with one-pass decoding.

Two results constrain the design. One-Step Diffusion Policy reports that a 1.49 Hz diffusion policy fails completely on a moving-target pick (0%), while its 62.5 Hz student succeeds every time. This tells us dynamic tasks must be in the evaluation, not just static success rates. RDT-1B and Discrete Diffusion VLA both report that dense visual tokens can drown out language. Routing language only through the slow pathway could worsen or relieve this, so we measure it directly.

Research Program

We start from a trained teacher policy and keep its tokenizer. The teacher is a discrete-token vision-language-action model of the kind used by Discrete Diffusion VLA or FASTer. The model has three parts.

Slow pathway. This is the teacher's own vision-language backbone. It runs on the images and the instruction every $K$ control steps and writes a cache of context tokens.

Fast pathway. This is a compact image network, on the order of a small ViT or ResNet. It runs on the current frame and proprioception at every control step and is trained with

$$ \mathcal{L}_{\text{fast}} = \big\| g_\phi(o_t,\, c_{t-\tau}) - E(o_t) \big\|^2 , $$

where $o_t$ is the current observation, $c_{t-\tau}$ is the slow cache computed $\tau \le K$ steps earlier, $E$ is the frozen teacher encoder, and $g_\phi$ is the student. Conditioning on the cache lets the student learn the update to known semantics rather than re-derive them. This training needs only frames, never actions. We therefore train it on the union of the demonstrations, action-free play video, and rendered RoboCasa scenes. RoboCasa can produce frames essentially without limit.

Action decoder. A light bidirectional transformer attends to the cache and the fast tokens. It emits a full action chunk in one forward pass, trained with a Paraformer-style glancing sampler. It is fine-tuned on the demonstrations with the encoders frozen.

The analysis has three parts. First, we map success and reaction latency as functions of the refresh interval $K$, to find how stale the semantics can get before performance breaks. Second, we test language robustness under paraphrased instructions, since language now enters only through the slow path. Third, we measure the amount of action-free data needed for the fast encoder to match the teacher.

Baselines

The naive reference is the teacher run as published, re-encoding every chunk at full size. Its success rate is the ceiling we hope to match, and its latency is the floor we must beat.

The strongest published operating point on LIBERO is FASTer, with 97.9% success and 112ms latency on an RTX 5090. On SimplerEnv-Bridge it reports 87.9%. Discrete Diffusion VLA reports 96.4% on LIBERO at 14.53 Hz, 54.2% on Bridge, and 0.8% language degradation under paraphrase. Both are the strongest competitors because they already optimize the decoder. Any win we show must therefore come from perception. We will re-time both on the same RTX 5090 and on an embedded-class GPU, with identical camera inputs and fine-tuning data. "Beating" them means matching LIBERO success within about one point at no more than half the end-to-end latency, and significantly higher success on the perturbed tasks described below.

For reactivity, the reference is One-Step Diffusion Policy, with 84.3% on Robomimic at 62.5 Hz, which we will run on the same perturbed tasks. Two competitors get the same encoder-size budget as our fast pathway and the same demonstrations and tuning budget. One is a small encoder trained end-to-end on demonstrations. The other is the teacher with its visual tokens pruned to match our compute.

Three ablations isolate the key ingredients. The first removes the slow cache, giving a pure distilled small encoder. The second trains the fast encoder on demonstration frames only, which tests whether action-free data matters. The third replaces glancing training with plain parallel regression with a variational latent, as in ACT.

Why not just…?

Why not just use a smaller vision encoder and fine-tune it end-to-end? This is the most natural cheap fix, and it is a direct baseline. The evidence against it is indirect. Co-Learning and Modality-specific Distillation both find that teacher guidance lifts small models above their end-to-end versions. LIBERO shows that encoder choice strongly affects transfer (forward transfer of 0.70 for ViT-T vs 0.30 for ResNet-RNN on LIBERO-Object). But no reported result on Lacuna compares distilled and end-to-end small encoders inside a manipulation policy. If our second ablation, the demo-only student, matches the full method, then the data-scarcity argument fails and we will say so.

Why not just run the full model slowly and execute longer action chunks? Chunking does amortize encoder cost, and on static benchmarks it may be enough. The threat is real for LIBERO and SimplerEnv success alone. That is why our decisive test is dynamic. The 0% vs 100% gap in One-Step Diffusion Policy on a moving target shows that low perception rate fails when the scene changes mid-chunk. Our perturbed suite measures exactly this case. If long chunks match us there too, the proposal has no reason to exist.

Why not just distill the action head to one step, as in One-Step Diffusion Policy, and stop there? One-step action generation removes decoder latency but leaves the encoder, which FASTer identifies as the dominant cost. That approach is complementary to ours: the fast path's decoder could itself be one-step distilled. We run it as a baseline to quantify how much latency remains after the decoder is fixed.

Why not just keep the teacher and prune visual tokens? Token pruning still runs the full network depth on every frame. It also may cut exactly the fine spatial tokens that contact-rich motions need. We test it at matched compute.

Proposed Experiments

Static benchmarks. We evaluate on LIBERO (all suites) and SimplerEnv Bridge and Fractal against the numbers above. We report success, end-to-end latency, and encoder share of latency on both GPUs.

Perturbed variants. We build perturbed versions of LIBERO and Robomimic tasks in which the target object is moved by a scripted displacement at a random time mid-episode, mirroring the pnp-milk-move test. We report success and reaction latency, defined as milliseconds from the displacement to a measurable change in the commanded trajectory. No reported result was found on Lacuna for VLA policies under such perturbations, so all methods will be run by us.

Data scaling. On RoboCasa, we vary the amount of action-free rendered and play frames used to train the fast encoder, holding demonstrations fixed at 50 per task. We plot success against unlabeled frames. RoboCasa's reported gain from 28.8% to 47.6% with synthetic demonstrations is the reference trend, though that gain used action-labeled data.

Decisive checks. The first is whether success on the perturbed tasks rises as reaction latency falls, with the two-rate model matching the teacher at a refresh interval $K$ of several chunks. The second is whether paraphrase degradation stays near Discrete Diffusion VLA's 0.8% despite language entering only through the slow path. The third is whether the demo-only student falls clearly below the student trained on action-free data. Failure on the first check kills the reactivity claim, and failure on the third kills the data-scarcity claim.