Spatial audio models are usually trained on simulated rooms and then deployed on real microphones. One popular way to make them robust is to convert the microphone signals into spherical harmonics, the angular analogue of a Fourier series that describes the sound arriving from every direction around a point. Formats that keep only the lowest "order" of these harmonics (First-Order Ambisonics, or FOA: 4 channels) are blurry in angle. Higher orders (HOA: $(N+1)^2$ channels at order $N$) are sharper in principle. Every real-world sound localization benchmark in this area still reduces its 32-capsule recordings to FOA. The reason is not that higher orders are useless. It is that nobody knows whether they help or hurt once a simulation-trained model meets real hardware.
This note proposes to answer that question and to fix the likely failure. On a real spherical array, the higher-order coefficients are estimated by dividing out the sphere's scattering response, and that division blows up capsule noise and calibration errors at low frequencies. Above a certain frequency, the capsules are too far apart to represent the field, so different directions alias onto each other. Simulations that write out ideal spherical harmonic coefficients directly contain neither error. Our hypothesis is that the sim-to-real gap in this domain is concentrated in specific order-and-frequency cells, and that it grows with order when training data is ideal. We further hypothesize that the gap becomes flat in order, so that HOA beats FOA on real recordings, once the simulator produces raw capsule signals with realistic mismatch and passes them through the same encoder used on real data. We call this encoder-in-the-loop simulation.
Core Claim
The claim has two parts, and either can fail independently. The first is diagnostic. When a localizer is trained on ideally encoded simulated Ambisonics, its real-minus-simulated error should increase with truncation order $N$. The excess error should sit in the low-frequency, high-order cells, where noise is amplified, and in the high-frequency cells, where aliasing occurs. The second is corrective. If the simulator renders capsule signals on a rigid sphere, perturbs capsule gain, phase, and position by amounts typical of manufacturing tolerance, and encodes them with the real pipeline's regularized inversion filters, then a model trained this way should transfer better as $N$ rises from 1 to 4. FOA would then stop being the safe default.
This differs from the closest work in a specific way. Spatial LibriSpeech simulated a 19-channel spherical array and produced both FOA and HOA. It tested zero-shot real transfer only in FOA, reaching 12.43° median error on TUT Sound Events REAL, so it motivates the question but does not answer it. How to (Virtually) Train Your Speaker Localizer showed that adding measured array directivity to simulation helps real-world localization. It worked in raw microphone coordinates, never in the spherical harmonic domain, and did not ask how the realism a model needs changes with angular resolution. Spherical Harmonics Injection for Multi-Channel Speech Enhancement claims a 25-coefficient order-4 representation is hardware-agnostic, but it tested only on simulated data, where the transform is exact. Our diagnostic directly tests that claim on real arrays.
Research Program
The work has three components.
The first is a simulator for a 32-capsule rigid-sphere array with Eigenmike geometry. It uses geometric room simulation with frequency-dependent wall absorption and directional sources, following the realism recipe that Realistic sources, receivers and walls showed cuts real-world room-parameter error by over 70%. The simulator samples on the order of a thousand rooms, because Spatial LibriSpeech found room diversity matters more than hours of audio. It can emit either ideal spherical harmonic coefficients or raw capsule signals.
The second is a capsule-perturbation model with three ingredients: per-capsule gain and phase deviations, small position jitter, and self-noise. The raw signals are then encoded with the same regularized radial-filter encoder applied to real recordings, at a regularization level fixed in advance.
The third is a diagnostic. We train identical localizers at each order $N \in \{1,2,3,4\}$. We then measure the gap per order, and also per order-and-frequency cell by masking cells at test time. This produces a map of where simulation and reality disagree.
The localizer is held fixed across all conditions. It is a convolutional-recurrent network on spherical harmonic spectrograms with intensity-style input features, so that only the data pipeline varies.
Baselines
The naive reference point is an FOA model trained on ideal simulated Ambisonics. This is what the field actually ships, and it is the configuration behind the STARSS22 synthetic-pretraining result of 29.3° localization error and 21% macro F1 reported in STARSS22. The second baseline is the same model at orders 2 to 4, still trained on ideal coefficients. If the diagnostic hypothesis is right, this baseline gets worse with order on real data.
The strongest existing competitor is the realism-injection recipe of How to (Virtually) Train Your Speaker Localizer, which the authors report gives 4–7% higher real-world recall and about 2° lower mean angular error. We run it in its native raw-capsule form with measured directivity, on the same rooms, training steps, and hyperparameter search budget as our method.
The classical reference is steered-response-power and subspace localization on the same encoded signals. LOCATA reports top methods near 1.0° azimuth error on its single static source task, per The LOCATA Challenge. The file does not report a neural spherical-harmonic result on LOCATA's Eigenmike tasks, so no reported result found on Lacuna for that.
The ablation that isolates our key ingredient is a cheap substitute: add noise directly to ideal coefficients, with variance per order and frequency set to the analytic noise amplification of the radial filters. Comparing this against full capsule-level perturbation tells us whether structured, correlated encoding errors matter or whether a noise variance curve suffices.
Our method counts as beating the baselines on two conditions. First, at its best order it must lower real-recording error relative to the best of these baselines, by more than the seed-to-seed spread, on at least two of the real test sets. Second, its real-minus-sim gap must stop growing with order.
Why not just…?
Why not just stay with FOA? This is the real threat. STARSS22 found that FOA already beats raw tetrahedral microphone signals on real scenes, and FOA has the smallest noise amplification. If even encoder-in-the-loop HOA cannot beat FOA on real data, the honest conclusion is that FOA is the right operating point for simulation-trained models. Our order sweep reports exactly this, and we would publish it as the answer.
Why not just simulate more random rooms? Spatial LibriSpeech found that 10% of its rooms gave nearly the same result as all of them. RealMAN reports that ISM-trained localizers reach 52.4% accuracy within 5° on real recordings, versus 89.2% for real-trained ones. Room count is not the bottleneck. More importantly, room diversity cannot produce encoding errors, because those come from the array rather than the room.
Why not just add noise in the spherical harmonic domain? This might work as well, and it is cheaper. That is why it is our central ablation rather than a footnote. Uncorrelated noise cannot mimic aliasing, which leaks energy from one direction into another in a deterministic pattern, nor the cross-order coupling that capsule mismatch causes. If the analytic-noise ablation matches full capsule simulation, we will recommend it and drop the heavier pipeline.
Why not just fine-tune on real recordings? STARSS22 shows that real-only training collapses to 64.1° error because the data are small and class-imbalanced. We nonetheless report a fine-tuned variant of every condition, to check whether our pretraining advantage persists after adaptation.
Proposed Experiments
The primary real test bed is LOCATA, because it supplies raw 32-capsule Eigenmike recordings with optical motion-capture ground truth, which allows encoding at every order up to 4. We report azimuth and elevation error and recall within fixed angular thresholds on its static and moving-source tasks.
The second is STARSS22/23. The public release provides only FOA, which anchors the $N=1$ comparison against the published 29.3° figure. We will request the raw Eigenmike channels to extend the sweep there. If they are unavailable, we will say so rather than substitute another dataset.
A secondary test of the hardware-agnostic claim applies the order-4 projection to RealMAN's non-spherical 32-channel array. Its 52.4% sim-to-real accuracy is the naive figure to beat.
The decisive check is a plot of real-minus-sim error against order $N$, one curve per training pipeline. The hypothesis predicts a rising curve for ideal coefficients and a flat or falling curve for encoder-in-the-loop training, with the cell-masking map placing the ideal pipeline's excess error in the predicted noise-amplification and aliasing regions. If both curves rise, the spherical harmonic gap is not mainly an encoding artifact, and the proposal's mechanism is wrong. If both are flat, the ideal-coefficient worry was unfounded and HOA can simply be adopted.
References
- Spatial LibriSpeech: An Augmented Dataset for Spatial Audio Learning
- How to (Virtually) Train Your Speaker Localizer
- Innovative Directional Encoding in Speech Processing: Leveraging Spherical Harmonics Injection for Multi-Channel Speech Enhancement
- Realistic Sources, Receivers and Walls Improve The Generalisability of Virtually-Supervised Blind Acoustic Parameter Estimators
- STARSS22: A Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events
- The LOCATA Challenge: Acoustic Source Localization and Tracking
- RealMAN: A Real-Recorded and Annotated Microphone Array Dataset for Dynamic Speech Enhancement and Localization