Do Spoken Robot Policies Listen? Separating Hearing from Heeding Across Languages

Robots that take spoken commands are now built end to end. A speech encoder turns the audio into vectors, and those vectors sit beside camera images inside one model that outputs arm motions. These systems are trained and tested almost entirely in English. When someone speaks Cantonese or Polish to them, they will fail. The open question is why, because the reason decides the fix. One possibility is that the English-centred speech encoder cannot tell the instructions apart in another language. The other is that the encoder can still tell them apart, but the action model has learned to read the task mostly from the camera image and treats speech as a weak hint. In the second case it falls back on whatever task the scene suggests as soon as the speech looks unfamiliar.

We propose to measure these two failure sources separately. We call them the hearing gap (the encoder fails) and the heeding gap (the encoder succeeds and the policy ignores it). We then test whether the heeding gap can be closed by changing how the policy is trained, without changing the speech encoder. Visual occlusion is where the heeding gap should cost the most. When the gripper blocks the camera during the final contact phase, the scene stops supplying cues about what the task is. Only a policy that actually listened can still commit to the right goal.

Core Claim

We claim three things about speech-conditioned vision-language-action models (VLAs: models that map camera images plus an instruction to robot actions). First, a large share of their zero-shot failure on unseen languages is a heeding gap, not a hearing gap. Second, that heeding gap is a form of modality dominance: the network learns from whichever input gives the easiest early gradient and under-trains the rest, here vision over speech. Third, a training-time penalty that makes actions depend on the instruction, combined with visual masking shaped like real self-occlusion, closes much of the heeding gap while leaving the English-centred encoder untouched.

The claim is falsifiable with a simple check. We train a small linear probe on frozen speech-encoder features. If the probe can identify which task was asked for in Cantonese far more often than the policy actually performs that task, a heeding gap exists. If the policy's failures are mostly completions of a different feasible task, rather than random flailing, the gap has the shape that dominance predicts.

Where This Sits in Prior Work

Two systems motivate the question directly. VLAS feeds Whisper speech features, downsampled five times, alongside image tokens into a language-model backbone and reaches 94.2% initial-task success on CALVIN. ELLSA adds full-duplex listening and acting, with separate speech and action experts, and reaches 89.4% on LIBERO. Both are trained and evaluated only on English speech, and neither asks whether vision crowds out speech inside the shared backbone. We reuse VLAS as the test model and its CSI dataset as the training data. What we add is cross-lingual and occluded evaluation, plus an analysis of reliance on each input.

The speech literature tells us to expect a hearing gap. Towards Universal Speech Discrete Tokens reports that tokens from monolingual self-supervised models do not carry over to other phoneme sets. Zero-Shot Cross-Lingual Phonetic Recognition with External Language Embedding recovers 4–8% absolute phone error rate on five unseen languages (Cantonese, Lao, Croatian, Spanish, Polish) by adding language-family and phone-inventory metadata. Our work does not compete with these encoder-side fixes. It asks how much of the robot's failure would remain even if they succeeded perfectly. We adopt those five languages as our held-out test set.

The modality-dominance literature supplies the mechanism. What Makes Training Multi-Modal Classification Networks Hard? shows that modalities overfit and generalize at different rates. In that paper, naive audio-visual fusion (71.7%) falls below video alone (72.6%) on Kinetics, and its Gradient-Blending method fixes this (75.8%) using offline, per-modality loss weights. Agrawal et al. document the mirror image in visual question answering, where models answer from the question text and ignore the image. Are Audio-Language Models Listening? finds text dominance in audio-language models and corrects it at inference time by steering hidden states toward the difference between real audio and a silent input. That steering gains 8.05 points on the MMAU audio-understanding benchmark for Qwen2-Audio and 14.1 points on its speech subtasks, at the cost of a second forward pass. We borrow that paper's counterfactual logic: compare the model's behaviour with and without the real signal. We move it from inference time into training and from question answering into control.

The occlusion literature constrains our design. DECO, TacFiLM and ManiWAV all show that vision drowns out sparse contact signals and that fingers block the view at the decisive moment. They fix this by adding touch or contact sound through dedicated pathways. In TacFiLM, for example, tactile-conditioned feature modulation reaches 66% on unseen HDMI plugging where token concatenation gets 0%. None of these papers studies the instruction stream. We treat their finding as a prior: whatever vision dominates, it will dominate speech too. We use their occlusion setting as the stress test that strips away the visual crutch.

Research Program

The study has three parts.

Measurement. We build a cross-lingual extension of CSI. The CALVIN instruction templates are translated into the five held-out languages. Native speakers record a test set. Multilingual text-to-speech supplies any additional training audio; the file lists no specific system for this, so we name none. We then measure the two gaps separately on the frozen VLAS speech encoder.

  • Hearing ceiling. A linear probe is trained on English features to predict the task identity (one of CALVIN's 34 tasks) and is then applied to the other languages.
  • Heeding rate. This is the fraction of episodes in which the policy completes the instructed task.
  • Scene-default rate. This is the fraction of failures in which the policy cleanly completes some other task that is feasible in the scene.

CALVIN suits this test because every scene supports many tasks, so the image alone cannot determine which one is meant. A wrong-task completion is therefore direct evidence that the policy read the task from somewhere other than the speech.

Reliance analysis under occlusion. At each step we compute two counterfactuals. One replaces the speech with a mismatched instruction or with silence. The other masks the target region in the static and gripper cameras during the terminal phase, using masks rendered from the arm's actual geometry rather than random patches. The change in the action distribution under each counterfactual gives a per-step reliance profile for speech and for vision. Dominance predicts two things: speech reliance should collapse to near zero after the first few steps, and under occlusion the policy should drift toward scene-default behaviour.

Intervention. We fine-tune VLAS with two additions, using English training speech only, so that the five test languages stay unseen. The first is occlusion-shaped masking of terminal frames. The second is a counterfactual-instruction margin:

$$\mathcal{L} = -\log \pi(a \mid o, s) + \lambda \max\big(0,\; m - [\log \pi(a \mid o, s) - \log \pi(a \mid o, s')]\big).$$

Here $\pi$ is the policy's distribution over actions, $a$ is the demonstrated action, $o$ is the (possibly masked) observation, $s$ is the true spoken instruction, $s'$ is a spoken instruction for a different feasible task in the same scene, $m$ is a margin, and $\lambda$ is a weight.

The margin term requires the demonstration to be noticeably more likely under the true instruction than under a wrong one. A policy that ignores speech cannot satisfy it. The margin needs no new demonstrations and adds no inference cost, unlike inference-time steering. VLAS emits tokenized actions, so the log-likelihoods are available directly.

Baselines

Naive reference. VLAS as released, evaluated zero-shot on the five languages, with and without occlusion. VLAS reports 94.2% initial-task success on English CALVIN. No reported result was found on Lacuna for VLAS or any speech VLA on non-English instructions, so we establish that number ourselves.

Cascade. Multilingual speech recognition, then translation into English, then a text-conditioned VLA trained on the same CALVIN data. This is the strongest practical competitor for task identity. VLAS reports that it outperforms a cascaded ASR pipeline on CALVIN, but the file gives no cascade number. On VLAS's speaker-customization benchmark, the file reports 86.5% for VLAS against 19.2% for non-customized VLAs; that comparison concerns speaker identity, not language.

Multilingual data augmentation. VLAS fine-tuned on synthesized speech in additional training languages that are disjoint from the five test languages. This tests whether more linguistic variety alone teaches the policy to heed speech.

Dominance controls. Three methods, each run on the same data and the same fine-tuning schedule:

  • ModDrop, following Wolf et al., with random dropping of vision or speech at a tuned rate.
  • Gradient-Blending, with loss weights computed per modality from its overfitting-to-generalization ratio.
  • Specialist-guided steering at inference, adapted from Are Audio-Language Models Listening? to the speech layers of VLAS.

Ablations. Margin loss alone, occlusion masking alone, and random-patch masking in place of geometry-shaped masking.

Fairness and the bar to beat. All methods share the CSI training data, LoRA rank, total GPU-hours, and a 20-run hyperparameter search for each method. Beating the baselines means two things. The proposal must raise the average heeding rate across the five unseen languages above every non-cascade baseline while staying within one point of VLAS on English. It must also cut the scene-default rate under occlusion relative to ModDrop and Gradient-Blending.

Why not just…?

Why not just run multilingual speech recognition and translate to English text? This is a real threat, and we expect it to win on task identity for high-resource languages such as Spanish and Polish. Our claim does not depend on beating it there. The cascade still feeds a text VLA, and that VLA may also be vision-dominant. If so, it should show the same scene-default drift under occlusion, which our reliance analysis will test. Recognition is also weakest exactly where the phonetic work above shows the hearing gap is largest: Lao and Cantonese. If the cascade matches us on every language and under occlusion, we will report that end-to-end cross-lingual speech control is not worth pursuing yet.

Why not just swap in a more multilingual speech encoder? That attacks only the hearing gap. The probe-versus-policy comparison exists to show whether this is the binding constraint. If the hearing ceiling on unseen languages is already far above the heeding rate, a better encoder raises a ceiling the policy is not reaching. If the ceiling turns out to be near chance, the premise of this proposal fails and encoder work is the right direction.

Why not just use modality dropout or Gradient-Blending? Gradient-Blending computes fixed weights per modality offline. Its own results show that one weighting cannot track rates that differ, and robot phases flip which input is informative within a single episode. Dropping vision wholesale removes the geometry that action prediction needs. Dropping speech teaches the policy to cope without speech, which is the opposite of what we want. Both are run as baselines. Our ablations will show whether the margin term adds anything beyond them.

Why not just steer at inference time? Steering doubles forward passes. The inference-cost results reported for active viewpoint selection in ActiveVLA (0.53 s per step versus 0.26 s for a fixed view) show that per-step overhead of this size already matters for closed-loop control. Steering also amplifies whatever speech signal exists and does not teach the policy to use it. We include it as a baseline and report both success and latency.

Decisive Checks

The first check is diagnostic. On Cantonese, Lao, Croatian, Spanish and Polish, the probe ceiling must sit well above the zero-shot heeding rate, and wrong-task completions must make up most of the failures. If failures are mostly incoherent motions, the dominance story is wrong.

The second check is causal. The margin-plus-masking model must narrow the difference between probe ceiling and heeding rate without any non-English training speech. Masking alone must not do the same, which would show that the margin, rather than occlusion augmentation, is the active ingredient.

The third check concerns occlusion. Under arm-shaped terminal masking, speech reliance in the trained model should rise during the occluded phase. In VLAS it should stay flat while scene-default completions climb.

A secondary transfer check uses LIBERO-Goal, whose tasks share one scene layout but have different goals. Spoken instructions are synthesized in the same way, and ELLSA's 89.4% English LIBERO average serves as context, not as a target.