Audio language models are large text models with an audio encoder attached in front. They often answer questions about how something was said (is the speaker angry, whispering, a child) by reading what was said. If the words are "I'm thrilled" and the voice is flat and furious, the model tends to answer "happy." Probing studies show the vocal evidence is present inside the encoder and is then overridden downstream. We propose a training-free decoding rule that fixes this at answer time. The model scores each candidate answer three ways: from the real audio, from a delexicalized copy of the audio where the words are made unintelligible but pitch, loudness and voice quality survive, and from the model's own transcript given as plain text. It then rewards answers supported by the delexicalized audio and penalizes answers that the transcript alone would predict. The penalized quantity is not the model's unconditional language prior, which is what existing contrastive decoders remove. It is the specific shortcut "infer the speaking style from the words."
This matters for two reasons. The current fixes for this failure require fine-tuning, and every result so far is on clean or studio speech. Deployed systems hear street noise, radio channels and reverberation. In those conditions the acoustic cues get weaker and the text shortcut plausibly gets stronger, and nobody has measured this. Our second goal is to measure that coupling and to make the strength of the correction respond to it. The answer-time correction must also leave ordinary transcription intact, so the same decoding rule is tested on speech recognition word error rate as well.
Core Claim
The paralinguistic "modality shortcut" is a form of internal language model bias. Instead of the prior $P(Y)$ over word sequences that speech recognition work subtracts, it is a prior over answers conditioned on the lexical content. It can therefore be estimated and divided out at inference time, the way density-ratio fusion removes an estimated internal language model score in end-to-end recognition (Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition). The right "negative" for this bias is the transcript, not silence or injected noise. The right "positive" helper is speech with its words destroyed. Concretely, for a question $q$, audio $X$, the model's own transcript $\hat{T}(X)$, and a delexicalized copy $D(X)$, we score each answer token $y$ as
$$s(y) = \log P(y \mid X, q) + \lambda_a \log P(y \mid D(X), q) - \lambda_c \log P(y \mid \hat{T}(X), q),$$
where $\lambda_a$ sets how much the words-removed acoustic stream is trusted and $\lambda_c$ sets how much of the content-driven prediction is removed. The second novel element is that $\lambda_c$ and $\lambda_a$ are set per utterance rather than fixed. $\lambda_c$ scales with measured agreement between the full-audio and transcript-only answer distributions, which estimates how content-driven the model is behaving right now. $\lambda_a$ shrinks as the delexicalized stream's own confidence falls, which happens under heavy noise. We expect this to address the static-strength sensitivity reported for contrastive decoding in ASR.
Grounding and Relation to Prior Work
The motivating evidence comes from three diagnostic papers. Do Audio LLMs Listen or Read? builds VoxParadox, where transcripts assert the wrong attribute. It shows Audio Flamingo 3 and Qwen2-Audio mostly follow the text, and that playing audio backward raised paralinguistic accuracy. That last finding is the seed of our positive stream. Heard but Not Heeded finds 82–85% style-probe accuracy in the late encoder against 19–53% at the output. It also reports that Qwen2-Audio is 97.7% content-driven, which is the quantity our $\lambda_c$ term targets. Resurfacing Paralinguistic Awareness locates a sharp loss of speaker information around layer 7 of Qwen2.5-Omni and Kimi-Audio. All three work on clean speech.
Two of these papers also supply competing methods, and both require training: VoxParadox's prompt-conditioned layer mixer with preference optimization, and PE-FT's selective fine-tuning with an auxiliary head. We do not retrain. We expect our method to be complementary to theirs, not a replacement.
One result constrains the design. The gradient-reversal experiment in Heard but Not Heeded found style and content linearly entangled in the encoder. That rules out cheap linear disentanglement and is why we separate content from style in the input (delexicalization) and the output (transcript stream) rather than in hidden states.
The decoding machinery is borrowed. Whisper-CD contrasts clean-audio logits against noise-injected, silent and time-shifted inputs to remove hallucinations in long-form recognition. AVCD contrasts against attention-masked audio or video tokens and applies contrast only when output entropy is high. Both negatives elicit an unconditional or ungrounded prior. Our negative is a content-conditioned prior, which is the one that actually produces the VoxParadox errors. Both methods also assume the positive input is clean. Ours is evaluated precisely when it is not.
Large Language Models are Efficient Learners of Noise-Robust Speech Recognition (RobustGER) shows noise appears as diversity in N-best transcripts. We reuse that idea: transcript-stream uncertainty serves as one noise signal for setting the weights. RobustGER itself discards all prosody, so it cannot address our target tasks.
Research Program
Stage 1: measure the coupling between noise and the shortcut. We corrupt VoxParadox, the paralinguistic and perception subsets of MMSU (MMSU), and Expresso. Corruption comes from noise families the models are unlikely to have been tuned on: real CHiME-4 backgrounds, the eight NOIZEUS noise types at 0–20 dB signal-to-noise ratio, RATS-style radio channel distortion, and simulated reverberation. We test Qwen2-Audio, Audio Flamingo 3 and Qwen2.5-Omni.
At each condition we record three things. The first is task accuracy. The second is VoxParadox adversarial-label agreement, meaning how often the model picks the answer the transcript implies. The third is the content-prosody leakage metric from Heard but Not Heeded. We also re-run layer probes to see whether the encoder's late peak survives noise.
This separates two hypotheses. In the first, noise destroys the acoustic evidence: probes fall. In the second, noise mainly shifts reliance toward text: probes hold while adversarial agreement rises. Only the second regime is fixable by decoding. Stage 1 tells us how much of the problem is in reach.
Stage 2: compare delexicalizers. Reversal, the only delexicalizer with evidence so far, has a flaw: it inverts pitch contours, so a rising question tone becomes falling. That is fatal for MMSU's tone-direction tasks. We therefore compare reversal against three alternatives. Low-pass filtering near 400 Hz is a classic way to make speech unintelligible while keeping F0 contour and intensity. Syllable-length segment shuffling preserves the local contour but not the global one. Resynthesis of an F0 and energy track on a neutral carrier is the third option.
For each delexicalizer we measure two things. Lexical destruction is measured as Whisper WER on the transformed clean audio. Paralinguistic preservation is measured as probe accuracy on Expresso. We measure both again under each noise type, because low-pass filtering keeps low-frequency noise and stationary noise is roughly unchanged by reversal.
Stage 3: build and test the decoding rule. We implement the scoring rule above. The adaptive weights come from two quantities: the divergence between the full-audio and transcript-only answer distributions (setting $\lambda_c$), and the entropy of the delexicalized stream (setting $\lambda_a$). Following AVCD, we skip the extra passes when the full-audio answer is already confident. Cost is three forward passes per contrasted step.
Stage 4: check for cross-task damage. For transcription, the transcript stream predicts the transcript, so subtracting it would be destructive. We therefore require the rule to switch itself off in that case. Large agreement between the audio and transcript streams on an open-ended transcription prompt should drive $\lambda_c$ to zero, but this must be shown, not assumed. We report WER on LibriSpeech and CHiME-4 with the rule active everywhere. We also compare it against an explicit prompt-type gate in the spirit of the layer mixer.
Baselines
The naive reference is greedy or standard decoding of each base model with the benchmark's own prompt. For this baseline, VoxParadox reports 17.40% accuracy for Audio Flamingo 3 and 14.85% for Qwen2-Audio (Do Audio LLMs Listen or Read?). MMSU reports 60.57% overall for Qwen2.5-Omni-7B (MMSU). We re-run every baseline ourselves on identical audio, prompts and noise conditions, because none of these numbers exist under noise. For noisy VoxParadox and noisy MMSU on any model, no reported result was found on Lacuna.
The strongest existing method is PCLM plus preference optimization. It is reported to raise VoxParadox accuracy from 17.40% to 65.20% on Audio Flamingo 3 and from 14.85% to 72.30% on Qwen2-Audio. It also lifts Audio Flamingo 3's MMSU paralinguistic subset from 37.74% to 54.78% (Do Audio LLMs Listen or Read?). It is the strongest competitor because it targets exactly this failure on exactly these models. If released weights or training recipes allow, we re-run it on our noisy splits using the authors' training data. That tests whether a method fine-tuned on clean speech degrades in noise. Our method uses no training, so we cap its tuning budget at the same held-out development split used to select the authors' checkpoints.
Beating it on clean VoxParadox is not our bar; a training-free method may fall short there. Our bar is twofold. First, recover a substantial fraction of its clean gain at zero training cost. Second, match or exceed it on the noisy splits. Stacking our decoding on top of PCLM should give a further gain under noise.
The second strongest competitor is a direct port of Whisper-CD's multi-negative scheme (noise-injected, silent and shifted negatives combined by log-sum-exp) to the audio language models. We tune its strength on the same development split. Whisper-CD reports Whisper Large-v3 WER on CORAAL falling from 208.76% to 45.77% (Whisper-CD), which shows the machinery works for recognition. It has never been run on paralinguistic tasks. AVCD-style audio masking is included as a third contrastive baseline.
Two ablations isolate our ingredients. In the first, the transcript negative is replaced by the silence negative, which tests whether the content-conditioned prior is what matters. In the second, the adaptive weights are replaced by the best single static pair tuned on clean development data, which tests the noise-adaptive weighting. For speech recognition, the reference is each model's own WER with standard decoding.
Why not just…?
Why not just feed the delexicalized audio alone? This is the most serious threat. VoxParadox's reversal experiment already raised accuracy with no contrast at all; the file does not give the size of that gain, so no reported number is found on Lacuna. If delexicalized-only input matches our three-stream rule, the contrastive component is unnecessary and we will say so. We expect it to lose on tasks that need words and voice together, such as MMSU sarcasm and intent, or prosody-dependent meaning like the Korean wh-phrases in Prosody in Cascade and Direct Speech-to-Text Translation. On those tasks, removing the content removes half the evidence. We report "delexicalized only" as a full baseline, broken down by task type, so the experiment settles this directly.
Why not just reuse Whisper-CD or AVCD unchanged? Their negatives expose what the model says with no audio grounding. In VoxParadox, though, the wrong answer is grounded in audio: in the words. A silent or noise-injected negative does not reproduce "the transcript says happy," so it should not cancel that preference. The silence-negative ablation tests this. If it performs as well as the transcript negative, our central claim about which prior to subtract is wrong.
Why not just fine-tune with PCLM and preference optimization or PE-FT? On clean benchmarks, these probably win, and we do not claim otherwise. Their costs are training and model-specific data. They also come with no evidence under noise: Resurfacing Paralinguistic Awareness reports 97–98% paralinguistic-appropriate responses only on clean child-safety speech. Our noisy evaluation of their checkpoints, and the stacked condition, show whether the two approaches are substitutes or complements.
Why not just denoise the audio first? Enhancement removes non-stationary energy, and whispers and laughter look a lot like that. Qwen2.5-Omni's 93.9% whisper and 85.1% laughing accuracy (Heard but Not Heeded) are exactly what an enhancer could erase. We run an enhance-then-answer baseline with EDNet (EDNet), broken down per attribute. We expect it to help content-heavy tasks and hurt vocal-event tasks. If it helps everywhere, the noise half of our proposal is weaker than we think.
Decisive Checks
Four results would confirm or kill the idea.
First, under noise, adversarial-label agreement on VoxParadox should rise while late-encoder probe accuracy stays relatively stable. If probes collapse instead, decoding cannot help, and the project redirects toward encoder robustness.
Second, the transcript-negative rule should beat the silence-negative ablation on VoxParadox and on the MMSU paralinguistic subset at every noise level. Without this, the core claim fails.
Third, adaptive weights should beat the best static weights on noise families excluded from development tuning. We hold out RATS-style channels and two NOIZEUS noise types entirely.
Fourth, with the rule always on, WER on LibriSpeech and CHiME-4 should stay within a small margin of standard decoding. If it does not, the self-gating claim fails, and the method needs a prompt-type gate like PCLM's to be safe across tasks.
References
- Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition
- Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
- Resurfacing Paralinguistic Awareness in Large Audio Language Models
- Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding
- AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding
- Large Language Models are Efficient Learners of Noise-Robust Speech Recognition
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
- Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases
- EDNet: A Versatile Speech Enhancement Framework With Gating Mamba Mechanism and Phase Shift-Invariant Training