Recalibrating Activity-Coupled Sound Vectors Across the Simulation-to-Reality Gap

Systems that detect and locate sounds — "a door knock, 40 degrees to the left, starting now" — are mostly trained on simulated audio, because recording real rooms with ground-truth source positions takes motion-capture rigs and dozens of actors. A popular output format, the activity-coupled Cartesian direction-of-arrival vector (ACCDOA), folds both answers into one 3D arrow per sound class. The arrow's direction says where the sound is. Its length says how confident the model is that the sound is happening at all. A single fixed cutoff on that length, usually 0.5, decides detection. We propose to test a specific, cheap-to-check hypothesis. When these models move from simulated to real rooms, a large share of the collapse comes from the arrow's length drifting while its direction stays partly usable. If so, much of the loss can be recovered without any real labels. The fix is to re-estimate, per class and per room, what "active" looks like, using only unlabeled real audio and the temporal steadiness of the predicted direction.

This matters because the reported real-world numbers are poor, and the usual remedies all need something scarce. The STARSS22 paper reports that synthetic pre-training followed by real fine-tuning reaches only a 21% F-score at 29.3° localization error. Fixes that do better rely on labeled real data, measured room responses, or cameras. A label-free correction at the output layer would apply to any ACCDOA model in any new room.

Core Claim

We claim that under sim-to-real shift, ACCDOA vector lengths shrink and scatter in a class-dependent and room-dependent way. They shrink most for rare and reverberation-sensitive classes. As a result, the fixed threshold $\tau=0.5$ misses events whose direction estimates are still coherent. Two signals can replace that threshold without labels. The first is the per-class distribution of vector lengths in a target room. The second is how steady the predicted direction is over a short window. Real sources produce steady directions, while spurious activations wander.

For class $c$ in a window of $W$ frames ending at frame $t$, write $\mathbf{u}_{c,k}=\mathbf{v}_{c,k}/\|\mathbf{v}_{c,k}\|$ for the unit direction of the predicted vector $\mathbf{v}_{c,k}$. Directional steadiness is then the mean resultant length

$$ s_{c,t}=\Big\|\frac{1}{W}\sum_{k=t-W+1}^{t}\mathbf{u}_{c,k}\Big\|, $$

which is near 1 when directions agree and near 0 when they scatter. On unlabeled audio from a target room, we fit a two-component mixture over the pair $(r_{c,t}, s_{c,t})$, where $r_{c,t}=\|\mathbf{v}_{c,t}\|$. One component stands for "silent" and the other for "active." A frame is declared active when the posterior of the active component exceeds one half. A model-selection test, either the Bayesian information criterion or a likelihood-ratio test of one versus two components, decides whether class $c$ is present in that room at all. This keeps the mixture from inventing an active mode for absent classes. Directions are left as predicted. Only the detection decision changes, and that decision also gates which direction estimates are scored.

The difference from closest work is concrete. The ACCDOA papers (ACCDOA, ACCDOA) introduced the length-as-activity coupling and the fixed threshold, but evaluated on synthetic data, where the threshold is well matched. STARSS22 and STARSS23 measured the sim-to-real gap with multi-ACCDOA baselines. Both kept the output representation and its threshold untouched. Meta-SELD++ adapts the network to new rooms, but it needs labeled support clips in each room and does not examine vector lengths. We change no network weights and use no target labels.

Research Program

The program has three parts, and the first can end it.

Part 1: diagnose where the failure lives. We take ACCDOA and multi-ACCDOA models trained under three regimes and split their real-room error into a detection part and a direction part using oracle substitution. The three regimes are synthetic-only, synthetic plus augmentation, and synthetic pre-training with real fine-tuning.

  • Supplying ground-truth activity while keeping predicted directions shows how good the directions are when detection is perfect.
  • Supplying ground-truth directions while keeping predicted lengths isolates detection error.

We also plot per-class length histograms in simulated versus real rooms and measure how far apart they are. If the oracle-activity condition shows that directions are themselves badly wrong, then recalibrating length cannot help much. In that case the project redirects toward the spectral-resolution and room-fingerprint problem flagged by divided spectro-temporal attention. That paper found performance dropped when spectral resolution rose past $F'=16$.

Part 2: label-free recalibration. We build the mixture procedure above, operating per class and per room on unlabeled target recordings of a few minutes. For multi-ACCDOA, the calibration is applied after the usual step that merges tracks pointing within 30° of each other (Multi-ACCDOA). This avoids the duplicated-track targets of its training scheme, auxiliary duplicating permutation-invariant training, which would otherwise create spurious agreement between tracks.

The design deliberately avoids aligning internal network features with a domain discriminator. Adversarial spatial masking showed, for LiDAR, that such discriminators latch onto clean-versus-noisy shortcuts. For audio, they also risk distorting the phase cues that localization depends on. Working only on outputs sidesteps both problems.

Part 3: optional self-training. A second stage uses the recalibrated detections as pseudo-labels to briefly fine-tune only the shallow layers. These layers are chosen because Meta-SELD++ found deep layers most vulnerable to room-specific reverberation. This stage is reported separately, so that the zero-training result stands on its own.

Baselines

The naive reference is the model's output with the standard fixed threshold $\tau=0.5$, applied to each training regime. For the fine-tuned regime, the STARSS22 paper reports 21% F-score and 29.3° $LE_{CD}$ for its multi-ACCDOA baseline. The real-only floor is 11% F-score at 64.1°. We retrain this baseline ourselves on the same splits so that all comparisons share data.

The strongest audio-only competitor on STARSS23 is the divided spectro-temporal attention model with Single-ACCDOA and augmentation (DST). It reports a SELD score of 0.407, error rate 0.495, F-score 42.7%, and localization error 16.5°. On STARSS22, the strongest single system is EINV2 with augmentation chains (EINV2+aug), which reports $F_{20^\circ}$ = 42.4% and $LE_{CD}$ = 19.3° on validation. EINV2 uses separate tracks rather than ACCDOA, so we include it as a representation-level competitor.

We reproduce DST and apply our recalibration on top of it. Beating it means improving its F-score and SELD score on held-out rooms, with $LE_{CD}$ not worse by more than 1°. The comparison holds the network, training data, and tuning budget fixed, and the recalibration uses only unlabeled target audio. For the synthetic-only regime evaluated on STARSS23, we found no reported result on Lacuna, so we establish it ourselves.

Meta-SELD++ reports 21.6° $LE_{CD}$ on STARSS23, against 25.2° for its supervised model. Because it uses labeled support clips, we treat it as a labeled-adaptation reference, not a like-for-like competitor.

The two ablations that isolate the key ingredient are length-only mixture calibration, which drops $s$, and steadiness-only gating. Together they show whether directional steadiness adds anything beyond per-room length rescaling.

Why not just…?

Why not just tune per-class thresholds on a small labeled real validation set? This is the real threat. If a handful of labeled clips per room recovers everything, our contribution shrinks to the label-free setting only. We therefore run it as an oracle-style upper bound, giving it the same number of target clips with labels that we use unlabeled, swept from one to ten minutes. If labeled tuning with one minute of data matches labeled tuning with ten, a reviewer can fairly argue that a minute of annotation is cheaper than our method. We would report that honestly. The case for our method rests on the fact that STARSS-style labels require motion capture (STARSS22). A deployed device in a new room can collect unlabeled audio, but not ground-truth positions.

Why not just augment the synthetic data harder? Augmentation chains help: EINV2+aug lifted F-score from 32.3% to 42.4%. Yet that same system still scored 2.2% F-score on "Water tap," a class whose short, weak events are exactly where a shrunken vector length would fall below a fixed threshold. Augmentation and recalibration are not exclusive, so we apply recalibration on top of the augmented model. The per-class results on rare classes will show whether augmentation already removes the length drift.

Why not just fine-tune on real data? Fine-tuning needs labels, and even fine-tuned models face room shift at deployment. Fine-tuned models are one of our three training regimes, and the room-held-out split tests whether recalibration still helps them.

Why not just go back to separate detection and localization heads? A separately trained detection classifier may suffer the same confidence drift. Reverting also brings back the loss-weighting friction that ACCDOA removed while reaching 7.9° on synthetic data. We test this directly by training a two-head model and applying the same mixture calibration to its detection probabilities. If that model's confidences shift as much as ACCDOA's lengths, the problem lies in detection confidence generally, not in the coupled representation.

Proposed Experiments

Training data. Training sources are the DCASE 2021 synthetic set (TAU-NIGENS) and synthetic mixtures built from measured room responses, following EINV2+aug. We also include image-source simulated rooms in the style of IRS, so that measured and computationally simulated rooms can be contrasted.

Evaluation. Evaluation uses STARSS22 and STARSS23 in both FOA and MIC formats with room-held-out splits. Recalibration audio comes from unlabeled clips of the held-out room that are disjoint from the scored clips. Metrics are $ER_{20^\circ}$, $F_{20^\circ}$, $LE_{CD}$, $LR_{CD}$, and SELD score, with per-class F-score reported for rare classes. We also report the following diagnostics:

  • the oracle-substitution decomposition,
  • the distance between simulated and real length distributions,
  • how often the mixture's presence test makes false calls on classes absent from a room.

Decisive checks. Three outcomes would decide the project.

  1. In Part 1, oracle activity must reduce real-room error far more than oracle direction does. If not, the core claim is wrong.
  2. Label-free recalibration must close a substantial fraction of the gap between the fixed threshold and labeled per-class tuning, on synthetic-only, augmented, and fine-tuned models.
  3. Adding directional steadiness must beat length-only calibration, especially in high-reverberation rooms. That is where Meta-SELD++ reports its environment extractor struggles, and where we expect lengths alone to be least informative.