Does Hearing a Question Make an Expert Model Guess?

Speech-capable language models are starting to receive spoken questions about drug interactions, engineering calculations, and financial rules. Two facts from separate literatures suggest this may go worse than either literature predicts. The first comes from text-only work on prompt sensitivity, which studies how much a model's answer changes when the wording changes but the meaning does not. That work finds such flips are concentrated where the model is unsure of its answer, and that this happens most in technical domains (ProSA). The second comes from speech work, which finds that saying a question aloud instead of typing it changes the model's decisions even when the content is identical (Said Aloud, Read Different). Nobody has asked whether these are the same phenomenon. If they are, a spoken expert question is a worst case: the modality shift acts as one more wording perturbation, and it lands on exactly the questions where the model's knowledge is thinnest.

We propose to test that claim directly, and to test it on real human speech rather than only the machine-generated speech that current cross-modal studies use. The central measurement is simple. For each expert question, we record how confidently the model answers it in text. Then we ask whether that confidence predicts which questions flip when the same question is delivered by a text-to-speech (TTS) voice, and by human speakers reading or paraphrasing it. The contribution is a per-question account of spoken-expert failure. It separates three causes: the model misheard the domain term, the model never really knew the answer so any perturbation flips it, or something specific to audio beyond both.

Core Claim

We claim that, in expert domains, instability under spoken delivery is largely predicted by the model's text-mode answer margin. The answer margin is the gap in probability between the model's top answer and its runner-up when it reads the question as text. We further claim that the extra penalty of natural over synthetic speech concentrates on questions containing domain terminology. Humans hesitate on, mispronounce, or re-segment terms such as drug names, theorem names, and accounting line items, which TTS renders cleanly.

The first claim extends ProSA's link between low decoding confidence and prompt sensitivity from wording changes to modality changes. The second locates the synthetic-to-natural gap in a specific place. The prior studies that relied on synthetic speech flagged that reliance as their own main limitation (Said Aloud, Read Different, Hear2Act).

Both claims can fail cleanly. If spoken flips are uncorrelated with text margin, or if they are just as margin-driven in general-knowledge controls as in expert domains, the "knowledge gap amplifies modality brittleness" story is wrong.

Relation to Prior Work

The closest work is Said Aloud, Read Different. It compares text against TTS-spoken queries on M2CQA-S, a culturally grounded image question benchmark covering 18 MENA countries. Its Contrastive Instability metric is the fraction of question triplets answered inconsistently among those where the model got at least one right. It reports that this metric rises from 0.43 to 0.71 for Qwen2.5-3B in Arabic when moving from text to speech. That study motivates ours but differs in three ways. It uses only synthetic voices, it covers general cultural content rather than expert knowledge, and it measures instability in aggregate without asking which items flip or why. We reuse its metric, adapted to paraphrase sets, and add the per-item confidence analysis and human speech.

ProSA supplies the mechanism we test. It connects prompt sensitivity to flat output distributions and shows sensitivity rising on MATH and code. It is text-only, and it reports that one in-context example is the biggest stabilizer, a finding we must control for.

Two speech benchmarks constrain the design. MMSU uses authentic human audio and finds that over half of model failures are perceptual, with the best model at only 53.6% on phonology. This warns that "mishearing" may dominate "not knowing," so our design must separate the two. Hear2Act finds that feeding raw audio barely helps (optimal-solution rate 14.6% to 15.3%). Having the model first write out what it heard nearly triples performance (39.6%). We borrow this "write it down first" idea as a condition and test whether it survives paraphrase, which Hear2Act did not check.

AHELM shows that a transcribe-then-answer pipeline (GPT-4o-mini Transcribe feeding GPT-4o) ranked 6th of 17 systems, ahead of several native audio models. This makes the cascade a serious competitor rather than a strawman.

On the domain side, TheoremQA and the pharmacist-exam work behind NLPEC (Towards Medical Machine Reading Comprehension) establish that models lack the governing knowledge in these domains, not just fluency. They supply our expert question pools. SQuAD-SRC supplies a general-domain control with real accented human recordings; it reports a 17.0-point F1 drop from text to audio.

Research Program

We build a small paired benchmark in which every item exists in five forms. The first is the original text. The second is text paraphrases, three per item, following ProSA-style template categories. The third is TTS renderings from two engines, at clean and noisy signal-to-noise levels. The fourth is human read-aloud recordings. The fifth is human spontaneous paraphrase: a speaker reads the question silently and then asks it aloud in their own words.

The spontaneous condition carries the natural disfluencies, false starts, and cross-phrase restructuring that SpokenWOZ identifies as what breaks models trained on clean input. We will verify semantic equivalence of spontaneous paraphrases with two annotators and drop disputed items.

The expert pools draw on two sources. The first is roughly 600 multiple-choice NLPEC pharmacist-exam questions. These are dense in drug names that are hard to pronounce, and their fixed options make answer margins directly readable from output probabilities. The second is roughly 300 TheoremQA items selected because they can be spoken without reading out long formulas. The control pool is 600 SQuAD-SRC questions, which already have multi-accent human audio.

Speakers will include native and non-native speakers of the question language. NLPEC is in Chinese, so its questions are recorded in Mandarin by Mandarin speakers; TheoremQA and SQuAD-SRC are English. This lets accent be crossed with domain.

For each model and item we compute three quantities. The first is the text-mode answer margin. For multiple choice, this is the gap in log-probability between the top option and the runner-up. For numeric TheoremQA answers, it is the agreement rate across 10 samples. The second is a binary flip indicator per spoken form: did the answer change from text mode. The third is term-level recognition. Each item has annotated domain terms, and we ask the model to transcribe the audio, scoring whether each term was recovered.

The main analysis is a per-item logistic regression. It predicts spoken flips from text margin, the paraphrase-flip rate in text, term-recognition failure, speech source (TTS versus human read versus human spontaneous), and domain. The claim holds if three things are true: margin and text paraphrase-flip rate carry most of the explained variance in expert pools, that share is larger than in the SQuAD-SRC control, and the human-minus-TTS penalty is concentrated on items with term-recognition failures.

The models are Qwen2.5-Omni-7B, Qwen3-Omni, and Phi-4-multimodal, which are the open systems used in the closest prior study. Where API access allows, we add one proprietary audio model. We use temperature 0 and a semantic answer checker to avoid the formatting artifacts that A Composable Evaluation System shows can swing scores by up to 80 points under rule-based parsing. Its 0.6B verifier (85.0% accuracy against humans) serves as the checker for numeric and free-form answers, with a human-checked subsample.

Baselines

The naive reference point is text-mode accuracy and text-mode paraphrase instability on the same items. Spoken instability that does not exceed text paraphrase instability means the audio channel adds nothing. No reported result was found on Lacuna for spoken versions of NLPEC or TheoremQA. For orientation in text only, TheoremQA reports GPT-4 with program-of-thought prompting at 51.0% and open models below 15%, and the NLPEC paper reports KMQA at 61.8% accuracy against a 60% pass line.

The strongest existing method is the transcribe-then-answer cascade, which AHELM found competitive with native audio models. We implement it by pairing a speech recognizer with the same backbone text model used inside each audio model where possible, so knowledge is held constant and only the listening path differs. All systems share identical prompts, decoding settings, and a fixed budget of three prompt templates, with no per-system prompt tuning. Our proposal is not a system that must beat the cascade on accuracy. It is a claim about where failure comes from. It "beats" the cascade explanation if the cascade still shows margin-predicted flips on items whose domain terms it transcribed correctly.

Two further conditions isolate the key ingredient. One is a gold-transcript condition, giving the audio plus the correct written question, which removes mishearing as a cause. The other is a Hear2Act-style condition in which the model first writes out what it heard and then answers. If flips on low-margin items persist under the gold transcript, knowledge fragility rather than perception is doing the work.

As a scale reference, we compare instability across the 3B, 7B, and 30B members of one model family. Said Aloud, Read Different reports that scaling from 3B to 30B reduces but does not remove instability. Our question is whether the reduction tracks the rise in text margin.

Why not just…?

Why not just use a speech recognizer and a text model, and treat the problem as recognition error? This is the real threat, and the design is built to settle it. If the cascade with correct term transcription shows no excess flips over text paraphrases, the "modality as perturbation" claim collapses to "recognizers miss jargon." That would still be a useful, narrower finding about term-level recognition. The gold-transcript condition plus the term-recognition covariate decides this per item. Hear2Act's evidence that textualizing helps points in the cascade's favor, which is why we test the cascade seriously rather than dismissing it.

Why not just use TTS with noise and accent augmentation instead of recording humans? It is far cheaper, and if it reproduced human-speech flips item by item, we would recommend it. We test this directly. For each item, we check whether noisy multi-voice TTS predicts the human read and spontaneous flips. The prior evidence that clean synthesis misses disfluencies and paralinguistic detail (SpokenWOZ, SpeechCaps) suggests TTS will match read speech better than spontaneous speech. That gap is the measured quantity, not an assumption.

Why not just measure text prompt sensitivity and assume speech behaves the same? Part of our hypothesis is that text paraphrase sensitivity does predict spoken flips. If it predicts them fully, the recommendation becomes "audit spoken expert deployments with cheap text paraphrase tests," which is worth knowing. What text-only testing cannot show is the residual: flips on high-margin items with correctly heard terms. The regression reports that residual explicitly.

Why not just add a few examples to the prompt? ProSA finds one example is the largest stabilizer, so we run a one-shot arm. The FinLMEval study (Is ChatGPT a Financial Expert?) found examples sometimes hurt specialized-domain accuracy, and TheoremQA found that supplying theorem definitions did not rescue open models. We therefore expect one-shot prompting to reduce formatting flips but not knowledge-driven ones. If it removes margin-predicted spoken flips too, that result replaces our proposed routing recommendation.

Decisive Checks

Three outcomes decide the proposal. The first is whether text margin predicts spoken flips more strongly in NLPEC and TheoremQA than in the SQuAD-SRC control. This is the amplification claim, reported as a difference in explained variance with bootstrap intervals over items. The second is whether flips on low-margin expert items survive the gold-transcript condition. This separates not knowing from not hearing. The third is whether the human-minus-TTS penalty concentrates on items with term-recognition failures and grows from read to spontaneous speech. This locates the synthetic-to-natural gap.

A positive result yields a concrete deployment rule. Before answering a spoken expert question end-to-end, check the model's text-mode margin on its own transcript, and route low-margin items to clarification. We will evaluate this rule by the fraction of wrong spoken answers it catches at a fixed abstention rate. A negative result on the first check, with a positive result on the third, would redirect the field toward domain-term recognition as the main target.