When people talk, stress often decides who a pronoun points to. Take "John called Bill a liar, and then he insulted him." Read with flat stress, "he" is John. Stress both pronouns ("then HE insulted HIM") and the listener flips the roles: Bill insulted John. The words are identical, so a system that reads only the transcript cannot tell the two readings apart. That matters now because audio language models (large language models that take raw speech as input) are increasingly used as automatic judges that score spoken agents. If a judge decides from the words and ignores the voice, it will reward agents that guess the default referent and penalize agents that listened correctly.
We propose a benchmark built from minimal pairs. Each pair is one spoken passage recorded twice, with the two recordings differing only in where the stress falls, and the correct referent flips between them. We use the pairs to measure whether audio judges change their verdict when only the stress changes, and whether their explanations admit that stress was the reason. We then separate two ways a judge can fail: it may not perceive the stress at all, or it may perceive the stress but not use it to decide reference. That split determines the fix. A perception failure calls for better acoustic front ends; an inference failure calls for changes to how the language model uses what the encoder already carries.
Core Claim
Our claim is that current audio judges will perform near chance on stress-flipped reference pairs, and that most of the failure happens after perception. Specifically, probes on the audio encoder will recover which word carries the stress well above chance, while the judge's final verdict ignores it. If this holds, reference is a case of the "encoder-output gap" (information present inside the model but missing from its output) that matters for downstream evaluation, not just for labeling emotion. The claim is falsifiable in two directions. If probes cannot recover stress placement, the problem is perceptual. If a text judge given gold stress marks also fails, the problem is referential reasoning and has nothing to do with audio.
What Exists and Where This Differs
Three results motivate the gap. Heard but Not Heeded probes Whisper, Qwen2-Audio and Qwen2.5-Omni on short expressive utterances from Expresso. It finds 82–85% probing accuracy for speaking style in late encoder layers but only 19–53% at the output, and it estimates that Qwen2-Audio is 97.7% content-driven. Do Audio LLMs Listen or Read? builds VoxParadox, where the transcript contradicts the voice (for example, happy words spoken angrily). Qwen2-Audio and Audio Flamingo 3 score 14.85–17.40% there, and a layer-mixing module plus preference tuning raises this to 65–72%. The Silent Judge shows that text judges shift verdicts by up to 30% on injected metadata cues while acknowledging those cues 0.0% of the time.
All three stop short of our question. The first two test isolated attribute classification (emotion, age, pitch), not reference in multi-entity discourse. The third covers text only.
The closest prior work on each edge constrains the design. Using prosodic annotations to improve coreference resolution of spoken text shows that hand-labeled pitch accents help a classical coreference system on German radio news. The gain is small, CoNLL F1 rising from 47.93 to 48.88, and it holds only for short noun phrases. This tells us to control noun-phrase length and to expect modest effects in natural speech, which is why our main instrument is controlled minimal pairs rather than found speech. That paper also used gold labels and no neural or judging component. ParaS2S is the nearest speech-judge work. It shows that end-to-end audio judges correlate poorly with humans (Pearson r ≈ 0.618) and that a decoupled pipeline correlates better (r ≈ 0.7–0.8). In that pipeline, an acoustic analyzer extracts labels and a text model scores the response. ParaS2S judges emotional and social fitness, not reference, and never asks where in the model the lexical shortcut arises. The Role of Prosody in Spoken Question Answering finds that spoken QA models barely use prosody when words are available: F1 over answer frames (FF1) is 36.0 on natural speech versus 33.4 on monotone speech. We reuse its signal manipulations, pitch flattening and 300 Hz low-pass filtering, as controls.
On the referential side, Removing the Training Wheels supplies the notion of mention density. Quiz Bowl text averages 3.7 mentions per sentence against 2.1 in OntoNotes, and newswire-trained systems fail on it. We use density as a difficulty axis.
Research Program
The benchmark crosses three factors. The first is the stress device: contrastive stress on pronouns, deaccenting of a given noun phrase that marks it as coreferent, or a nuclear accent (the main stress of the phrase) marking a new referent. The second is referential density: two candidate referents, or four to five candidates at Quiz-Bowl-like density. The third is speech source: controllable-emphasis text-to-speech, or recordings by human speakers.
The speech-source factor addresses a tension in the file. SMT-9B reports that synthetic speech works as well as human speech for disambiguation, while MMSU argues synthetic speech is unrealistically clean. We will report every result separately by source.
We target roughly 600 passage templates, each with two stress readings. Each reading is validated by at least five human listeners, and only pairs where listeners agree on the flipped referent are kept. For each reading, a candidate agent answer names a referent. The judge is asked whether the answer is correct, in both absolute and pairwise formats.
The primary metric is prosody-conditioned accuracy on the flipped member of each pair. This is the reading whose referent departs from the text default, so a transcript-only judge scores at most chance on it by construction. The secondary metric is a prosodic verdict flip rate: how often the verdict changes between the two readings when it should. This mirrors the verdict-shift measure in The Silent Judge. We also measure a stress acknowledgment rate, the fraction of correct flips whose justification mentions emphasis. Finally, we run a nuisance test: the same answer spoken confidently versus hesitantly should not change the verdict. This asks whether audio judges also carry the unacknowledged voice shortcut that the Silent Judge found for metadata.
For diagnosis, we train linear probes at every encoder layer, after the projector that maps audio features into the language model, and at several language-model layers. The probes predict two things: which word carries the stress, and which referent is intended. This speaks directly to the disputed location of information loss. VoxParadox places paralinguistic signal in encoder layers 5–15 and calls the projector a bottleneck. Heard but Not Heeded reports a late-encoder peak and argues the projector preserves information that the decoder then discards. We will test the same question on a task where the answer affects a verdict. The models are Qwen2-Audio, Qwen2.5-Omni and Whisper-based cascades. We include Gemini-1.5-Pro and GPT-4o-Audio as black-box judges without probing.
Last, we measure consequences. We score a set of spoken agents on the benchmark with each judge and ask how often the judge ranks a stress-heeding agent below a text-default agent. This quantifies reward misranking of the kind that would matter if such a judge were distilled into a reward model, as ParaS2S does.
Baselines
The do-nothing reference is a transcript-only judge. Whisper transcribes the passage and GPT-4o judges the answer from text. On flipped readings it should sit at or below 50% accuracy, because the text supports only the default referent. Any audio judge that scores no better than this is not listening.
The strongest existing competitor is a ParaS2S-style decoupled judge adapted to reference. An automatic pitch-accent detector marks stressed words in the ASR transcript, for example by capitalizing them, and a text language model reasons over the marked transcript. This is the strongest competitor because ParaS2S shows decoupling beats end-to-end judging (r ≈ 0.7–0.8 versus 0.618 on ParaS2SBench), and because stress placement is a discrete, easily transcribed cue. No reported result was found on Lacuna for any decoupled or end-to-end judge on prosody-dependent coreference.
We also run the same text model with gold stress marks from the recording script. This gives an inference ceiling that removes perception from the picture.
The strongest method for making end-to-end models heed audio is the layer-mixing plus preference-tuning approach from VoxParadox (65.20% for Audio Flamingo 3 and 72.30% for Qwen2-Audio on VoxParadox). We reimplement it on the same backbones and train it on its original attribute tasks. Then we test whether it transfers to stress-dependent reference without referential training data.
To keep comparisons fair, every judge sees identical audio and prompts, and prompt engineering gets the same budget of trial prompts on a held-out development split. Any tuned component (the accent detector, the layer mixer) is trained on data disjoint from the benchmark templates.
The ablations isolate the key ingredient, stress. Pitch-flattened versions of each reading should push a genuinely stress-using judge back to chance. Low-pass-filtered versions keep stress while destroying the words, which tests whether the stress is perceived at all. An end-to-end judge counts as beating the decoupled baseline only if it matches or exceeds it on flipped-reading accuracy and also collapses under pitch flattening. The collapse is what shows it is not exploiting duration or text-to-speech artifacts. Human listener accuracy on the validated pairs is the target.
Why not just…?
Why not just use ASR plus a strong text judge? The minimal-pair construction answers this. Both readings produce the same transcript, so the cascade gives the same verdict on both and is wrong on one of them. Prosody in Cascade and Direct Speech-to-Text Translation makes the same point for Korean question words, whose meaning depends on pitch that transcription discards. We include this cascade as the floor rather than argue against it.
Why not just add stress marks to the transcript and let a text model reason? This is the real threat, and we treat it as a result rather than an obstacle. If predicted stress marks plus a text judge approach the gold-mark ceiling and human accuracy, the conclusion is clear. The failure of end-to-end judges is then a design problem that decoupling fixes, and practitioners should decouple, as ParaS2S found for emotion. The benchmark and the probing diagnosis remain the contribution in that case. If the approach falls short, that points to weak accent detection, especially on dense, long noun phrases, where Rösiger and Riester saw prosody stop helping. The gap between gold and predicted marks settles which case holds.
Why not just prompt the audio judge to attend to emphasis, with step-by-step reasoning? MLLM-as-a-Judge reports that step-by-step reasoning can hurt multimodal judges, because they invent details and then rationalize from them. The Silent Judge shows that justifications hide the cues actually used. We run this prompting as an arm under the same prompt budget. We check whether any accuracy gain survives pitch flattening, and whether acknowledgment of stress in the reasoning predicts correctness.
Why not just fine-tune with the VoxParadox recipe? It may work, and we test it directly. That recipe trains the model to report acoustic attributes, which is different from using stress to bind a pronoun to an entity. Transfer to the flipped readings, especially at high referential density, is the test.
Decisive Checks
The central result has three parts: flipped-reading accuracy of end-to-end judges, the gold-mark text ceiling, and stress-probe accuracy by layer. Low judge accuracy together with high probe accuracy and a high gold-mark ceiling supports the inference-failure claim. Low probe accuracy points to perception instead. The density factor tests whether referential difficulty compounds the shortcut, which we expect to show as a larger gap between judges and humans at five candidates than at two. The human-versus-synthetic split tests whether conclusions drawn from text-to-speech carry over to real speech. Finally, the confident-versus-hesitant manipulation, reported with its acknowledgment rate, tests whether audio judges carry the unacknowledged shortcut behavior documented for text judges.
References
- Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Using prosodic annotations to improve coreference resolution of spoken text
- ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction
- The Role of Prosody in Spoken Question Answering
- Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers
- Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
- Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark