Speech-capable language models reason worse when a problem is spoken than when the same problem is typed. SpeechR calls this the "speech tax." Nobody knows what causes it. In people, the cause of a similar penalty is well studied. Listeners hold the early parts of a sentence in a limited working memory until the grammar resolves. When the grammar is tangled, less capacity is left for decoding the sound itself. Uslar et al. found that object-initial sentences raised the noise level at which listeners could still recognize speech by up to 3 dB. This note asks whether end-to-end speech models show the same trade-off. The specific question is whether the speech tax grows with the distance between grammatically linked words, whether that growth is worse in noise, and whether it depends on how long those links stretch in audio time rather than in words.
The claim is testable and the answer matters either way. If the tax is mainly a structure-over-audio problem, it should concentrate on sentences with long dependencies. It should be larger in noise and larger when the same words are spoken more slowly. It should also be repaired by having the model restate the problem in short, simply structured clauses before it solves anything. If instead the tax is ordinary mishearing, it should add up independently of syntax, and a transcription front end should remove it. The design separates these two accounts with controlled minimal pairs rather than benchmark-wide averages.
Core Claim
The claim is that part of the speech tax is an interaction between grammatical integration cost and acoustic difficulty. It is not the sum of a perception error and a reasoning error. "Integration cost" here means how much a reader must hold in mind to connect grammatically dependent words. The measure is Dependency Locality Theory (DLT), the psycholinguistic metric that Syntactic Blind Spots applied to text math problems. That paper motivates this proposal and supplies its method. It showed that text LLM failures on GSM8K-style problems correlate with high DLT cost, and that offline rewrites lowering DLT recover many failures. It never touched audio.
Mechanistically, one spoken word spans many more encoder frames than one text token. Each dependency is therefore much longer in a speech model's input sequence than in the text sequence. If the model integrates structure across that span, then slowing speech down should hurt. Slowing lengthens every dependency in frames while leaving the words and their grammar unchanged. Human listeners typically benefit from slower speech, so a penalty from slowing would be a signature specific to the machine.
Research Program
The core artifact is a factorial stimulus set built from GSM8K, SVAMP, MultiArith and ASDiv, the datasets used by Syntactic Blind Spots. Each problem gets three versions with identical numbers and answer. The first is the original. The second is a DLT-reduced rewrite made with that paper's dependency-guided rephrasing. The third is a DLT-increased rewrite that introduces object-relative clauses, passives and fronted modifiers.
Semantic equivalence is checked in two ways. A strong text model must solve all three versions identically, and a sampled subset is checked by hand. Word count is held within a narrow band across versions so that DLT is not a proxy for length.
Each version is rendered to speech with several synthetic voices. It is then crossed with signal-to-noise levels, three speaking rates produced by pitch-preserving time-stretching, and two prosody conditions. The first prosody condition uses natural phrase boundaries. The second flattens boundary cues: pitch resets and lengthening before a boundary. This lets us test the unresolved question behind SpeechR's "acoustic invariance" finding. That finding is that models ignore prosody. The open question is whether models also fail to use boundary cues to parse hard syntax.
A smaller set of the same items is recorded by human readers. This guards against artifacts of the synthetic voices.
We evaluate models that take audio in directly. These are Qwen2-Audio-7B-Instruct (used in Audio-CoT), Qwen2.5-Omni-7B (the strongest open model on MMSU) and GLM-4-Voice-9B (the base of STITCH). Each model is also scored on the text form of every item, which gives an item-level speech tax.
The analysis is a mixed-effects logistic regression of correctness on the item's features, with random intercepts for items and voices:
$$\text{logit}\,P(\text{correct}) = \beta_m m + \beta_d d + \beta_s s + \beta_{md}\, m d + \beta_{mds}\, m d s + \beta_{mdr}\, m d r + \dots$$
Here $m$ indicates speech rather than text input, $d$ is the item's DLT cost, $s$ is noise level, and $r$ is speaking-rate stretch.
Two coefficients test the claim. The coefficient $\beta_{mds}$ captures whether syntax hurts more in noisy speech than it does in text. The coefficient $\beta_{mdr}$ captures whether syntax hurts more when dependencies are stretched in time.
The interventional arm tests whether the tax can be repaired at the level of structure. We call it structure-first listening. Before reasoning, the model is prompted to restate the spoken problem in its own text scratchpad as short clauses, one quantity or relation each, and then to solve from that restatement. It is also lightly fine-tuned to do this on CoT-Collection spoken data. This differs from Audio-CoT's description-first prompting, which asks the model to describe acoustic properties rather than to rebuild grammatical structure. It also differs from Syntactic Blind Spots, whose rewrite is done offline on text by a separate LLaMA-3 model. Our restatement is produced by the listening model itself, directly from audio. This reuses the text-scratchpad finding of Can Speech LLMs Think while Listening?: text reasoning is roughly twice as token-efficient as reasoning in speech tokens.
Baselines
The naive reference point is the audio model answering directly from speech with no reasoning prompt. Its gap to its own text-input accuracy on matched items defines the tax we try to explain.
The first real competitor is plain zero-shot chain-of-thought on audio input. It is the default fix, and Audio-CoT reports it raising MMAU accuracy for Qwen2-Audio-7B-Instruct from 55.60% to 57.80%, with losses on hard items. It is run with the same models, the same items and the same number of sampled answers as our method. Beating it means a larger recovery of the speech tax specifically in the high-DLT cells, not just overall.
The strongest competitor is a cascade. Whisper-class speech recognition feeds a text LLM of the same backbone size. On the transcript, the text LLM first applies the dependency-guided rewrite of Syntactic Blind Spots, then reasons. That paper reports a 15.48% accuracy improvement for Granite-7B on MultiArith from rephrasing. We found no reported result on Lacuna for this cascade-plus-rewrite pipeline on spoken math input, so we run it ourselves. The cascade is also the fairest test of the "mere mishearing" account, because it removes audio from the reasoning path entirely.
We also report STITCH-style chunked reasoning as an existing end-to-end reasoning method. On math datasets it beat no-reasoning baselines by 15% on average, per STITCH.
All systems get the same items, the same decoding budget, and the same fine-tuning data where any fine-tuning is done. Prompt tuning is limited to a fixed number of trials on a held-out development split.
The key ablation replaces the short-clause restatement with an unconstrained restatement of equal length ("repeat the problem in your own words"). Any gain that survives this swap is attributable to lowering structural cost, not to extra tokens or rereading. Structure-first listening counts as a success only if three conditions all hold. It must close more of the speech tax than zero-shot chain-of-thought and the unconstrained restatement. That advantage must be concentrated where $d$ is high. And it must match or beat the cascade in the noisy cells where recognition errors compound.
Why not just…?
Why not just use a cascade and forget end-to-end audio models? This is the real threat. If the cascade with a text rewrite erases the tax at every DLT level, the conclusion is that the tax is a transcription problem. Our interaction coefficients would then be near zero for the cascade. We would report that finding as the answer. To make the comparison diagnostic rather than a horse race, we stratify cascade errors by the word error rate of the recognizer on each item. If recognition itself degrades on high-DLT sentences in noise, the interaction lives in the recognizer. That would still extend Uslar et al.'s human result to machines, but it would redirect the fix to the front end. The cascade also discards prosody. The flattened-boundary condition checks whether end-to-end models gain anything from phrasing that a transcript cannot carry.
Why not just call it an input-length effect? High-DLT sentences could simply be longer or rarer. Word count is matched within the triples, so length is not the explanation. The time-stretch manipulation changes only frame length and leaves the grammar untouched. A pure length account predicts that slowing speech hurts equally at every DLT level. Our account predicts the harm grows with $d$, which is exactly what $\beta_{mdr}$ measures.
Why not just scale the model? Said Aloud, Read Different found that moving from 3B to 30B reduced but did not remove text-versus-speech instability. It reports Arabic contrastive instability for Qwen2.5-3B rising from 0.43 in text to 0.71 in speech. We include two sizes of the Qwen audio family to test whether $\beta_{md}$ shrinks with scale. We do not claim in advance that it will not.
Why not just prompt the model to paraphrase before solving? That is precisely our key ablation. Syntactic Blind Spots reports that its gains came from explicitly dependency-guided rewrites rather than generic rephrasing. If generic paraphrase works as well from audio, the structural part of our claim fails.
Proposed Experiments
The first decisive check is the regression on the synthetic factorial set, replicated on the human-read subset. The account survives only if $\beta_{md}$ is negative and $\beta_{mds}$ adds penalty beyond it in the direct-audio models, while the cascade's corresponding terms are near zero. The second check is the rate manipulation. A penalty from slowing that grows with DLT would be the clearest machine-specific signature. The third check validates on natural distributions: we regress the item-level tax on the procedural-reasoning split of SpeechR and on the GSM8K portion of SRQA against each item's DLT cost, without any rewriting. This tests whether the effect appears outside our constructed stimuli. The fourth check measures structure-first listening against the baselines above, cell by cell. The prediction is that its gains cluster in high-DLT, noisy, slowed conditions.
If time allows, the same logic can be turned to production. We would measure whether GLM-4-Voice's generated Mandarin speech loses tone accuracy as the syntactic complexity of its own spoken output rises. This would echo the non-native tone errors reported by Chen et al.. The scoring would use automatic tone recognition rather than the global naturalness score that STITCH reports. That arm is secondary. The perception result is the one that decides whether the syntactic-phonetic trade-off is a useful lens on why speech models reason worse.
References
- SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models
- Does sentence complexity interfere with intelligibility in noise? evaluation of the oldenburg linguistically and audiologically controlled sentence test (OLACS)
- Syntactic Blind Spots: How Misalignment Leads to LLMs’ Mathematical Errors
- Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
- STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models
- Can Speech LLMs Think while Listening?
- Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models
- iCALL corpus: Mandarin Chinese spoken by non-native speakers of European descent