Augmented-reality glasses and embodied assistants will increasingly have to answer a question that no current audio metric is built to check: where does this sound come from for that person? Suppose a system renders binaural audio (two-ear audio that carries direction cues) for a conversational partner or an avatar, rather than for the wearer. It must first carry out a mental rotation from the camera's point of view into the partner's. Vision-language models are known to fail at exactly this step and fall back on the camera's frame. When they do, the result is audio that sounds clean but points the wrong way. Our core claim is that these perspective errors fall into a few geometric symmetry classes: left–right mirroring, front–back flips across the axis between the ears, and elevation errors. Standard audio metrics, and even the best deep perceptual localization metric, are partly or fully blind to some of these classes. Whether the error matters to a human also depends on whether the listener can move their head.
We propose to build a controlled, human-rated stress set of such frame errors. We will use it to measure which metrics track human localization judgments for each error class and playback regime. We will also test a metric that makes the target listener's frame explicit and scores the full distribution of predicted directions, not a single angle. The work treats egocentric spatial reasoning as the source of the failures and treats the gap between objective metrics and human perception as the thing being measured.
Why this gap is real
Two separate literatures meet here.
On the vision side, models collapse when asked to take another viewpoint:
- Across 12 vision-language models, adopting an agent's viewpoint costs about 35% accuracy (Seeing Through Their Eyes).
- No model exceeds chance (about 29%) on a Three-Mountain-style rotation task (Vision Language Models See What You Want but not What You See).
- GPT-4o scores 34.98% on person-centric localization and often switches frames mid-answer (ViewSpatial-Bench).
These papers motivate the proposal: they tell us which errors to expect, mainly camera-frame answers given where agent-frame answers are required. They stay entirely visual. On the audio side, Can Large Language Models Understand Spatial Audio? and Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos reason about direction only in the frame of the recording sensor. Neither asks how a scene sounds to someone else.
Metric divergence supplies the second half of the argument. RIPPLE showed that independent per-channel phase recovery leaves waveform $L_2$ error looking fine while spatial localization drops to chance (57.3° polarization error, the random-guessing level). This is a clean example of a metric being blind to direction. DPLM built a deep perceptual localization metric from the hidden activations of a direction-of-arrival network (a network that estimates which direction a sound arrives from). On the P1 Speech listening data it reaches Spearman 0.94 with human ratings, against 0.03 for the classical binaural model BAMQ. However, DPLM is evaluated for horizontal angle around a fixed receiver, and the file lists elevation as an open problem. That boundary is exactly where front–back and elevation frame errors live. Because of this, DPLM both supplies part of our method and constrains what we can claim.
The cone of confusion explains why the symmetry classes matter. A source reflected across the plane through the ears produces nearly identical interaural time and level differences. A front–back flip therefore leaves time and level based metrics almost unchanged. Static listeners without their own head-related filtering (the direction-dependent spectral coloring from a person's ear shape) also confuse front and back frequently. Listeners who can turn their head resolve the confusion. As a result, whether a metric "should" penalize a front–back flip depends on how the audio will be played back. We know of no prior evaluation that conditions on playback regime in this way.
Core Claim
Perspective errors in rendered spatial audio fall into symmetry classes that existing metrics register unevenly. A metric that expresses directions in the target listener's frame, keeps the front–back and elevation ambiguity of its direction estimate, and is checked separately against head-tracked and static listening will track human judgments of these errors better than reference-based signal metrics and better than single-angle direction error. Perceived audio quality has to be reported on a separate axis, because a frame error leaves quality intact.
Research Program
The stress set. We render scenes in SoundSpaces 2.0 on Matterport3D meshes, the same toolchain used to build DSS LibriSpeech in Can Large Language Models Understand Spatial Audio?. Speech sources come from LibriSpeech, and multichannel training material for the direction estimator comes from Spatial LibriSpeech. Each scene has a camera and one or two human agents with known head poses. For each agent we render the correct binaural audio at that agent's head pose. We then render controlled corruptions:
- a camera-frame render, which is the perspective-locking error;
- a pure left–right mirror;
- a pure front–back flip across the interaural plane;
- elevation offsets;
- small azimuth offsets that serve as a graded control;
- a RIPPLE-style per-channel phase-recovery corruption, which gives a quality-preserving but direction-destroying case of a different kind.
We also include naturally occurring errors. These come from a perspective-locked pipeline: a vision-language model is asked for the source's direction relative to the agent, and its answer drives the renderer. This keeps the stress set tied to the failures that real egocentric reasoning systems produce.
The listening test. Listeners hear each clip in two regimes. In the static regime, the binaural clip is played as-is with non-individualized filters. In the head-tracked regime, the clip is re-rendered as the listener turns their head. Listeners report perceived direction on a sphere and give a MUSHRA-style quality rating (a standard listening-test protocol in which listeners score several versions of a clip against a reference). The quality protocol follows SEBASS as used by MAPSS. This yields a per-clip human localization error and a quality score in each regime.
The proposed metric. We train a direction-of-arrival network on binaural renders that include elevation variation and pairs of head rotations. Instead of a point estimate, it outputs a probability distribution $p(\theta)$ over directions $\theta$ on the sphere. Let $R$ be the rotation that maps world coordinates into the target agent's head frame, and let $d^*$ be the true source direction in world coordinates. The localization score is
$$ S = \mathbb{E}_{\theta \sim p}\big[\angle(\theta, R\,d^*)\big], $$
the expected angle between sampled directions and the true direction expressed in the listener's frame. In the head-tracked variant, the network sees the audio at two or more head yaws and combines its distributions. This mirrors how head motion breaks the front–back confusion.
As in DPLM, we also compute $L_1$ distances between the network's hidden activations for the output clip and for the correctly rendered reference, so that the metric can run with or without a reference. Quality is reported separately with MAPSS Perceptual Match, which measures self-distortion and artifacts.
Baselines
The naive reference points are full-reference signal metrics computed against the correct agent-frame render: waveform $L_2$, SI-SDR, and interaural time and level difference errors. These are what the egocentric audio work in the file reports, for example SI-SDRi in Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos. We expect $L_2$ and SI-SDR to penalize every corruption roughly equally, regardless of whether humans can perceive it, and we expect time and level difference errors to stay near zero for front–back flips.
BAMQ is included as the classical binaural model. It has a reported Spearman of 0.03 with human ratings on P1 Speech in DPLM.
The strongest existing perceptual competitor is DPLM itself, with a reported Spearman of 0.94 with human ratings and 0.86 with true angular distance on P1 Speech. We will retrain it with its published recipe on the same Spatial LibriSpeech-derived training data and the same compute and tuning budget as our network. This keeps the comparison about the design, not the data.
The strongest simple competitor is single-angle direction error: run a strong direction estimator on the output, rotate its point estimate into the agent's frame, and report the angle to the true direction. The relevant reference figures are a 6.60° median 3D localization error for the Spatial LibriSpeech 3D-CNN (Spatial LibriSpeech) and a 2.70° mean angular error for the LLM-based localizer (Can Large Language Models Understand Spatial Audio?). Both were measured on multichannel Ambisonic audio, not binaural audio. We found no reported result on Lacuna for any metric's agreement with human judgments on front–back or elevation frame errors.
The proposal beats these competitors if its rank correlation with human localization error is significantly higher, by bootstrap over clips and listeners, on the front–back and elevation classes in at least one playback regime. It must also not be worse on the azimuth classes.
The ablations remove one ingredient at a time:
- replace the distribution with its peak, which reduces the metric to the single-angle baseline;
- drop the head-rotation input in the head-tracked variant;
- drop the frame rotation $R$, so directions are scored in the camera frame.
The last ablation should make camera-frame renders look correct. That would confirm that the frame-conditioning is what catches perspective-locking errors.
Why not just…?
Why not just compute direction error with a good estimator and a rotation? This is the real threat. For left–right mirrors and graded azimuth offsets it will probably match our metric, and we expect to report that. It may fall short on front–back flips. A point estimator either confidently commits to the wrong lobe of the cone of confusion or splits the difference between lobes. In the static regime, humans share that ambiguity, so a confident penalty would over-penalize. In the head-tracked regime, humans resolve it, so a confused estimate would under-penalize. The peak-only ablation tests this claim directly. If it ties the full metric on every class and regime, we will drop the distributional component and report the stress set and the class-by-regime findings as the contribution.
Why not just use DPLM as it is? DPLM already shows that deep features from a direction network beat classical binaural models by a wide margin. Its validation, however, is horizontal-angle similarity around a fixed receiver. It never conditions on whose head the audio is meant for, and elevation is listed as unmodeled. The stress set measures exactly how far its 0.94 human correlation carries over to the front–back and elevation classes. If it carries over, DPLM plus an explicit frame rotation is the answer, and the proposal shrinks to an evaluation study.
Why not just fix the upstream reasoning, for instance with chain-of-thought prompting or a symbolic pipeline? Chain-of-thought prompting is inconsistent for perspective-taking (Seeing Through Their Eyes). Symbolic projective layouts in the style of Keep it SymPL push allocentric accuracy to 97.33% on its distance tasks. Even so, a pipeline that is right most of the time still needs a metric that detects the cases where it is wrong. Head-pose estimates and renderers also introduce their own front–back errors. Better reasoners reduce the frequency of frame errors but do not remove the need to measure them, so we use them as error sources in the stress set rather than as alternatives to it.
Proposed Experiments and Decisive Checks
The first check is the blindness profile. For each corruption class, we report the rank correlation of each metric with human localization error, separately for static and head-tracked playback. The prediction is specific: $L_2$ and SI-SDR will not distinguish perceptible from imperceptible corruptions, interaural time and level errors will miss front–back flips, and the static and head-tracked rankings of metrics will differ on front–back flips. If the two regimes give the same ranking, the playback-dependence part of the claim is wrong.
The second check is the two-axis dissociation. Phase-collapse and camera-frame renders should score high on MAPSS Perceptual Match while scoring poorly on human localization. This would show, in a perspective-taking setting, the quality-versus-direction split that RIPPLE found for $L_2$.
The third check is the natural-error test. We apply each metric to the renders driven by the perspective-locked vision-language model and ask which metric ranks a set of systems, such as end-to-end prompting and a SymPL-style geometry pipeline, in the same order as human listeners. Agreement with human system rankings on realistic errors is the outcome that would justify adopting the metric. Correlation on synthetic corruptions alone would not.
References
- Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models
- Vision Language Models See What You Want but not What You See
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- Can Large Language Models Understand Spatial Audio?
- Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos
- RIPPLE: Generating Multi-Channel Phase, Not Recovering It
- DPLM: A Deep Perceptual Spatial-Audio Localization Metric
- Spatial LibriSpeech: An Augmented Dataset for Spatial Audio Learning
- MAPSS: Manifold-based Assessment of Perceptual Source Separation
- Keep it SymPL: Symbolic Projective Layout for Allocentric Spatial Reasoning in Vision-Language Models