Voice assistants on smart glasses are now scored by other language models. A model acting as a judge (an "LLM judge") reads the assistant's answer and decides whether it was good. In the main wearable benchmark, that judge reads only text: a transcript and a reply. It never hears who was speaking, or from where. This matters because the hardest cases for these devices are spatial. A bystander talks while the wearer does too, and the assistant must answer the wearer and ignore the bystander. A reply that fluently answers the bystander can look excellent to a judge that only sees a mixed transcript. If that happens, the evaluation loop rewards exactly the failure that multi-microphone hardware exists to prevent.
We propose to measure this blind spot and close it cheaply. The fix is to give the judge a spatially attributed scene transcript: a short text record of who said what, and from which direction and distance. It is built from the device's multi-channel audio before the judge sees anything. The central claim is that a structured text record of this kind recovers nearly all the judging accuracy lost on spatially dependent cases. If that holds, frontier text judges can stay in place, and an end-to-end audio judge would not be needed. The study is built so the claim can fail. We compare against a judge that directly ingests learned spatial audio tokens, and that judge may win when speech overlaps.
Core Claim
WearVox built an egocentric benchmark of 3,842 multi-channel recordings from AI glasses and scored outputs with a text-only Llama 3.3 70B judge, reporting r = 0.89 against human ratings. It also showed that multi-channel input raises side-talk rejection from 85.4% to 93.9%. These two facts motivate our question. The correlation was measured over the whole output distribution, which is dominated by cases where the addressee is obvious. It says little about the minority of cases where only the sound field can tell you whom the assistant should have answered.
We claim three things. First, text-only judges fail silently on wrong-addressee responses, meaning replies that would be correct for a bystander's utterance. Second, judges given explicit direction labels will be swayed by those labels even when the labels are wrong, and their written rationales will not mention them. This follows the pattern The Silent Judge found for recency and provenance cues, with a 0% cue acknowledgment rate. Third, a scene transcript estimated from the audio itself, not taken from oracle metadata, closes most of the gap.
What is new is not "a multimodal judge." It is a controlled test of what evidence about the sound field a judge needs, and a demonstration of which representation delivers it.
Research Program
The first component is a contrastive test set built on WearVox, which ships distance and angle metadata for conversational partners and bystanders. For each recording that contains a bystander, we pair the original reference-quality reply with a generated addressee-swapped reply. The swapped reply is fluent, correct, and of similar length, but it answers the bystander's utterance. Matching length guards against the verbosity bias documented in MLLM-as-a-Judge. Humans with access to the separated audio and the metadata label a sample to confirm that the swap is genuinely wrong. They also rate overall quality, so we can compute human correlation on this harder slice and not only on the full benchmark.
For moving-talker conditions we create additional mixtures from LOCATA. LOCATA documents degraded tracking and voice-activity "ghost tracks" with moving talkers, which lets us stress the estimated scene transcript under realistic localization error.
The second component is the scene transcript itself. It is produced by a multi-channel front end that does three things: separates or beamforms each active source, transcribes it, and tags it with an estimated direction, a near/far label, and a self-voice flag. The result is rendered as a few lines of text that the judge prompt can include.
We build this in two variants. One uses a classical localizer plus beamformer and an off-the-shelf recognizer. The other uses a geometry-agnostic encoder in the style of PhaseCoder. PhaseCoder converts multi-channel audio and microphone coordinates into tokens a language model can read, and it reports 7.44° localization error on LOCATA. We reuse its encoder as a tool for producing transcripts, not as a judge.
The third component is a cue-injection audit modeled on the Silent Judge method. We deliberately corrupt the scene transcript in three ways: flip directions, mislabel which talker is the wearer, and mark the louder talker as primary. We then measure how often verdicts change, and whether the judge's rationale ever cites the corrupted field. This tells us whether a judge that improves with spatial text is reasoning over it or simply obeying it.
Baselines
The naive reference point is the WearVox protocol unchanged: a text judge (Llama 3.3 70B, plus GPT-4o for cross-checking) reading a mono transcript and the reply. Its reported figure is r = 0.89 with human ratings on WearVox (WearVox). To beat it, a method must keep at least that correlation on the full benchmark while substantially raising the rate at which addressee-swapped replies are ranked below correct ones.
The oracle ceiling is the same judge given ground-truth WearVox direction and distance metadata, formatted exactly like our scene transcript. It shows how much of the gap spatial information can close at all. A method that comes close to this ceiling using only estimated information is the result we want.
The strongest competing method is a spatial-token audio judge: a PhaseCoder-style Gemma 3n model fine-tuned with LoRA to output pairwise verdicts directly from spatial tokens and audio. This is the obvious "multimodal judge" answer and the most serious threat. As an end-to-end system it may handle overlapping speech better than any text intermediate. A cautionary number is PhaseCoder's real-world speaker-attribution accuracy of 52.73%, which suggests attribution from tokens alone is far from solved. For fairness, this judge and our pipeline share the same front-end encoder weights where applicable, the same human-labeled training and development split, and a matched budget of GPU hours and tuning runs. All frontier text judges receive identical prompts and a fixed number of prompt iterations.
A mono "thinking" judge (Gemini 2.5 Flash Thinking given the mono audio) represents the view that reasoning substitutes for spatial evidence. As a task model on WearVox it reaches up to 91.4% side-talk rejection.
Two ablations isolate the key ingredient. In the diarization-only ablation, the scene transcript separates speakers but removes direction, distance, and self-voice fields. This asks whether "who said what" alone suffices. In the shuffled-direction ablation, the fields are kept but permuted across talkers. This tests whether the judge actually uses the spatial content or only benefits from the extra formatting.
Why not just…?
Why not just give the text judge the ground-truth metadata? Real deployments and new recordings have no ground truth, so an evaluation method that needs it cannot score field data or self-generated training data. We still run this condition as the oracle. If estimated scene transcripts fall far below it, the bottleneck is the front end and not the judge, and the paper reports that honestly.
Why not just train an audio judge on spatial tokens end to end? Two findings point the other way. MLLM-as-a-Judge found that vision-language judges often judge better from text descriptions than from raw pixels. PhaseCoder's 52.73% real-world attribution accuracy suggests token-level attribution is fragile. Against this, SING found that explicit numerical direction prompts were inferior to learned continuous spatial embeddings for its task model. STAR-Bench found that caption-only inputs lose 35.2% on spatial tasks. Both are real threats. Our response is the head-to-head comparison above, broken down by amount of temporal overlap. We expect the token judge to win only in heavy overlap, and we will report it if it wins more broadly.
Why not just let a thinking model infer the addressee from language? Linguistic cues often reveal side-talk, which is why mono thinking models do well on WearVox. Addressee-swapped replies are constructed to be linguistically plausible, however, so semantics alone should not separate them. WearVox also reports thinking-model time-to-first-token sometimes exceeding 5 seconds, which makes them expensive judges at scale. The mono thinking baseline on the contrastive set settles this directly.
Why not just use a cascade of beamformer, recognizer, and text judge? That cascade essentially is our classical variant, and AHELM found such modular pipelines competitive, ranking 6th of 17 systems. We embrace the simple design. The contribution is showing which fields the judge needs, how corruptions in those fields propagate silently into verdicts, and whether a learned encoder improves the transcript enough to matter.
Proposed Experiments
The decisive experiment is pairwise accuracy on the WearVox contrastive set: how often each judge prefers the correct-addressee reply over the swapped one, compared across all baselines. We report it alongside human correlation on the full benchmark and on the bystander slice. Our claim holds if three things are true. The naive judge must be near chance on swapped pairs while keeping high overall correlation. The estimated scene transcript must recover most of the distance to the oracle. The diarization-only and shuffled-direction ablations must fall clearly below the full transcript.
The second experiment repeats this on LOCATA-derived moving-talker mixtures. It plots judge accuracy against front-end localization error, which tells practitioners how accurate the localizer must be before judging stays trustworthy.
The third is the cue-injection audit. For each corruption type we report how often verdicts shift and how often rationales acknowledge the corrupted field, for both the text judges and the spatial-token judge. A finding that verdicts move under corrupted directions while rationales stay silent would show that spatially grounded judges inherit the Silent Judge failure. That would argue for auditing them with consistency checks on the scene transcript, not trusting their explanations.
For context on the open-ended spatial reasoning that motivates the work, no reported result was found on Lacuna for LLM-judge agreement on spatial-audio responses. Establishing that number is itself an output of this study.
References
- WearVox: An Egocentric Multichannel Voice Assistant Benchmark for Wearables
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
- The LOCATA Challenge: Acoustic Source Localization and Tracking
- PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs
- SING: Spatial Context in Large Language Model for Next-Gen Wearables
- STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence
- AHELM: A Holistic Evaluation of Audio-Language Models