Does the Voice or the Words Set the Values? Causal Tests of Language-Triggered Bias in Speech LLMs

Multilingual language models change their moral and political positions depending on the language they are prompted in. The same model scores as more authoritarian when asked in Mandarin than in English, and it switches from outcome-based to duty-based moral reasoning when a dilemma is posed in Urdu. Speech-capable models make this problem more tangled. A speech-capable model takes spoken audio in through a self-supervised speech encoder, which is a network pretrained on unlabeled audio that turns waveforms into vectors. Those vectors carry the words, but they also carry which language is being spoken, what accent the speaker has, and how the speaker sounds. We therefore do not know which of these signals actually moves the model's values when a question arrives by voice.

This note proposes to find out with interventions rather than observation alone. We keep the spoken content fixed and change the signals around it: which language the audio sounds like, the accent, and whether the input is natural speech or synthetic speech. We then surgically remove the "which language is this" information from the audio vectors before they reach the language model, and we measure how much of the cross-lingual value shift disappears. The sharpest test uses Hindi and Urdu. The two are nearly identical when spoken but are written in different scripts, and prior text studies report divergent moral behavior between them. Speech lets us ask whether that divergence belongs to the language or to the script.

Core Claim

Our hypothesis is that the value shift in speech models comes from two separable sources. The first is a content route: the model effectively reads the words, much as it would read a transcript. The second is an acoustic language-identity route: a low-dimensional "this sounds like language X" signal that the speech encoder passes forward, which on its own pushes the model toward that language's value profile.

Write $V(\cdot)$ for a scalar value score, such as the authoritarianism score on the F-scale questionnaire. Write $s_L$ for spoken input in language $L$, $t_L$ for the same item as text in $L$, and $\tilde{s}_L$ for the same speech after language identity has been erased from the encoder output. The quantity we care about is the fraction of the speech-induced gap that erasure removes:

$$ \rho_L = \frac{V(s_L) - V(\tilde{s}_L)}{V(s_L) - V(t_{\text{En}})} $$

A value of $\rho_L$ near 0 means the shift travels through content alone. In that case a speech model behaves like a text model reading a transcript. A value of $\rho_L$ well above 0, under the condition that comprehension of the item is preserved, means the model's values are set partly by how the input sounds. That would be a new and uncomfortable finding: accent and dialect could shift a model's ethics independently of what was said.

Relation to Prior Work

The motivating evidence is all text-only. High-Dimension Human Value Representation in Large Language Models embeds model answers across 25 languages and finds that expressed values cluster by prompt language more than by model. Democratic or Authoritarian? reports that Llama 4 Maverick scores 2.79 on the F-scale in English and 3.86 in Mandarin, against a midpoint of 3.5. One Model, Many Morals finds framework switches, including Urdu prompts eliciting deontology, and noisier ethics in low-resource languages. Language Model Alignment in Multilingual Trolley Problems shows models flipping decisions across 107 languages. We reuse their instruments. None of them can separate script, token identity, and meaning, because in text these three always arrive together.

The closest existing work is ACID Test, which evaluates cultural safety of audio-language models across 10 languages. Two features of its design limit what it can tell us. First, all of its audio is synthetic speech generated from machine-translated text. Second, it scores only end-to-end outputs. It therefore measures that spoken prompts yield culturally varying behavior, but it cannot tell whether the variation comes from content or from sound. We differ on both counts: we intervene inside the model, and we compare natural speech against synthetic speech.

Evaluating Bias in Spoken Dialogue LLMs holds text fixed and varies voice attributes, finding a decision unfairness score (GUS) as high as 0.21 in open models. It is English-only and targets demographic fairness rather than values. We borrow its "hold content constant, vary the voice" design and move it into the cross-lingual setting.

Two further papers supply mechanism. When Audio and Text Disagree shows that audio-language models' hidden states linearly encode audio-text conflicts with up to 98% accuracy, even though the language model ignores that signal in its outputs. This cuts against our hypothesis: it suggests the language model may simply ignore acoustic information and read content. We take the prediction seriously, and $\rho_L \approx 0$ is a live outcome. Exploring wav2vec 2.0 on Speaker Verification and Language Identification shows that language and speaker identity concentrate in lower encoder layers, which tells us where to look.

The intervention itself comes from LEACE. LEACE is a closed-form linear projection that makes a concept linearly unreadable while distorting the vectors as little as possible. On text, it cut a gender gap from 0.198 to 0.084 at roughly a 2% accuracy cost. It has not been applied to speech representations.

Research Program

We work with modular open speech-language models where we can reach the vectors passed from the speech encoder to the language model. These are BLOOMZMMS (Teaching a Multilingual LLM to Understand Multilingual Speech), which pairs a frozen MMS speech encoder with a frozen BLOOMZ language model; encoder–language model pairs assembled in SLAM-LLM, such as HuBERT or WavLM with an instruction-tuned backbone; and Qwen2-Audio and SALMONN from the ACID and MCR-BENCH evaluations. The modular models matter most, because their backbone can also be queried with text directly. This gives a clean within-model comparison of text, transcript, and speech input.

Stimuli. We build a spoken value battery from three sources: the 30 F-scale statements, a stratified subset of MultiTP trolley vignettes, and ACID Set-A items in its overlapping languages. Each item is rendered three ways. The first is text. The second is synthetic speech from MMS-TTS, which matches ACID's pipeline. The third is natural speech recorded by native speakers of English, Mandarin, Hindi, Urdu, Arabic, and Bengali. We also record accented English: the English items read by native speakers of each of the other languages. This crossing of content language with speaker background is what lets us separate "which language the words are in" from "which language the voice sounds like."

Erasure. We fit LEACE on FLEURS utterances with language labels (FLEURS). We fit it at two places: the adaptor output that feeds the language model, and a lower encoder layer. The fitted projection is then applied at inference to the value battery.

Hindi–Urdu test. Hindi and Urdu are the natural stress test. FLEURS reports only 51.7% language-identification accuracy on South Asian languages, and spoken Hindustani is largely shared across the two. We record the same colloquial dilemmas once and label them as either language. We then ask two questions. Does the Urdu-versus-Hindi moral gap that appears in text persist for acoustically indistinguishable speech? And if it persists, does it survive erasure?

Baselines

Written text in the same language. The naive reference point is the model's value score when the item is given as written text in the same language. The text-language gap defines what "cross-lingual bias" means for each model. For Llama 4 Maverick, the cited F-scale gap is 2.79 in English versus 3.86 in Mandarin, from Democratic or Authoritarian?. We compute the corresponding gap for each of our backbones ourselves with identical prompts and sampling settings.

Cascade on the model's own transcript. The strongest competitor for explaining speech behavior is a cascade: the speech is transcribed, and the transcript is fed to the same backbone as text. If speech-input scores match cascade scores, the acoustic route contributes nothing, and our hypothesis fails. To keep the comparison fair, we use the model's own speech recognition output, or an external recognizer for models without one, and the same decoding settings. We report transcription error so that we can separate content errors from acoustic effects. Beating this baseline means showing a statistically reliable difference between speech and cascade scores on items whose transcripts are correct.

Existing speech benchmark. The strongest existing method on the evaluation side is the ACID protocol. ACID reports only that no model exceeded 0.50 on its conversational score and that MERaLiON scored 0.35–0.46 on relevance. There is no reported result found on Lacuna for F-scale or MultiTP scores under spoken input, so our speech numbers will be the first on these instruments.

Ablations. Three ablations isolate the key ingredient. The first erases a random subspace of the same rank, which controls for generic distortion. The second erases speaker identity instead of language identity, which checks that we are not simply removing voice information. The third applies LEACE to language-model hidden states instead of audio vectors, which tests whether the language signal is created in the encoder or recovered later by the backbone. We give every erasure the same fitting data and the same rank budget.

Comprehension guardrails. Erasure only counts if the model still understands the item. We therefore report FLEURS character error rate, against the BLOOMZMMS figure of 12.4, and SpeechXNLI accuracy, before and after erasure.

Why not just…?

Why not just transcribe the speech and debias the text model? A cascade answers our question only if the acoustic route is empty, and that is exactly what we test. Beyond Content shows that prosody carries signals that transcripts drop, with 91.56% accuracy on sarcasm detectable only from paralinguistic cues. That is evidence that the channel can carry meaning. Even so, if our speech and cascade scores coincide, we will report that result and recommend cascades for value auditing. The experiment settles the question directly.

Why not just translate every spoken prompt into English first? Translating to English does produce invariance, but it does so by imposing English values on everyone. That is a policy choice, not an explanation, and prior text work documents that English carries its own tilt. The alternative is also silent on the Hindi–Urdu case and on accented English, where no translation is involved. We include translate-to-English as a mitigation comparison, but not as an answer to the attribution question.

Why not just run ACID-style behavioral evaluation with more languages? Output-level evaluation confounds content, script, and sound, and ACID Test cannot separate them by design. Only the crossed accent conditions and the erasure intervention can assign the shift to a route.

Why not just prompt the model to answer neutrally? Prompting is weak evidence in this setting. When Audio and Text Disagree found that telling a model to prioritize audio raised accuracy only from 1.5% to 6.9%. Prompting also changes the input, which confounds the attribution. We include a neutral-instruction condition only as a secondary mitigation arm.

Decisive Checks

The first result that matters is the speech-versus-cascade difference on correctly transcribed items. If that difference is null across models, we stop claiming an acoustic route.

The second is accented English. If English items read by Mandarin-native speakers shift F-scores toward the Mandarin text score relative to native-English readings, voice alone moves values. The FairDialogue voice-attribute effects suggest this is plausible, but they are measured on a different axis.

The third is $\rho_L$ under language-identity erasure. The erasure must beat both the random-subspace and speaker-erasure controls while keeping character error rate within a few points of baseline. Without that retention, any drop in the value shift could simply reflect lost comprehension.

The fourth is the Hindi–Urdu result. If the text moral gap between Hindi and Urdu collapses under shared speech, the "Loss in Low-Resource" and framework-switch findings of One Model, Many Morals are at least partly script artifacts. If the gap persists and erasure removes it, the model is assigning values to how a voice sounds. Either outcome is informative, and the design makes each of them observable.