A pair of earbuds or hearing aids hears a conversation with two microphones, one at each ear. In principle that is enough to answer "who said what, from where." In practice the two channels drift apart: a wireless link adds a clock offset, an earbud sits loose, or a hand or hat muffles one side. Mammalian hearing handles the same problem in adulthood. When one ear is plugged, adults learn to lean on the intact ear's spectral shape cues, and with feedback they also re-tune their sense of interaural timing and level. Adult myelin, the insulation that sets how fast nerve fibres conduct, thins measurably after deprivation, which changes conduction timing. We propose a single model that transcribes overlapping speech, labels speakers, and reports each speaker's direction as one interleaved output sequence. We then use the internal agreement among those output streams as the feedback signal that lets the model recalibrate its own two-channel front end after deployment, without labels.
The two-sentence version is this. Current joint transcription-plus-diarization models ignore spatial hearing, and current binaural separators ignore transcription and break when one channel is corrupted. We join them, and we show that the joint output makes possible something neither half can do alone: a label-free "adult recalibration" of interaural timing and level offsets, kept separate from a fast switch that decides which cues to trust right now.
Core Claim
The claim has three parts. First, serialized output training (SOT), which flattens overlapping speech into one token stream that interleaves timestamps, speaker tokens, and words, can be extended with discretized azimuth tokens computed from binaural input. Second, the redundancy in that stream can drive test-time correction of persistent channel distortions. A speaker token that persists across a turn implies a smoothly moving source. Interaural time and level differences must agree with each other and with monaural spectral cues for the same azimuth. Third, recalibration works best when it is split, as the biology suggests, into a slow parameter update and a fast trust gate. The slow update re-estimates per-device interaural delay and per-band gain, analogous to myelin-set conduction timing. The fast gate shifts weight between interaural and monaural spectral features as the acoustic context changes.
The biological evidence motivates the split. Behavioral training promotes multiple adaptive processes following acute hearing loss showed that adult humans under a unilateral earplug both reweight cues and remap them. Remapping required feedback, and in ferret cortex the two processes were uncorrelated across neurons, which suggests separate mechanisms. Context-Specific Reweighting of Auditory Spatial Cues following Altered Experience during Development showed the brain keeps the normal spatial map alongside the adapted one and switches between them by context, with no loss when the plug is removed. Sound-Evoked Activity Influences Myelination of Brainstem Axons in the Trapezoid Body showed that adult deprivation thins myelin and that modelling predicts about 30% slower conduction. Timing parameters are therefore plastic, activity-driven, and slow in adults.
We do not claim to model myelin biophysics. We borrow three of its properties for one module: it is slow, it is driven by activity statistics, and it acts on timing.
Relation to Prior Work
Unified multi-task speech models supply the decoding method. SOMSRED serializes timestamps, text, and speaker tokens in one decoder. It reduced 3-speaker diarization error rate (DER, the fraction of time assigned to the wrong speaker or missed) from 32.3% to 15.7%. It also found that averaged intermediate decoder features identify unknown speakers far better than learned speaker tokens. Unified Multi-Talker ASR with and without Target-speaker Enrollment adds a removable conditioning encoder. Separator-Transducer-Segmenter streams turn tokens, and it reports that extra latency penalties made training unstable. End-to-End Speaker-Attributed ASR with Transformer reaches 11.9% cpWER (concatenated minimum-permutation word error rate) on LibriCSS from one channel.
All four are monaural. We reuse SOMSRED's serialization and its intermediate-feature speaker embeddings, add binaural input and azimuth tokens, and avoid STS's latency penalties.
Binaural separators define the spatial side and constrain our design. Online Binaural Speech Separation Of Moving Speakers With A Wavesplit Network injects interaural phase and level differences into a causal separator through feature-wise modulation, which scales and shifts intermediate features. It jointly predicts direction of arrival, and it reduces speaker swaps from 3.4 to 0.6 on 24-second recordings with online clustering. Binaural Angular Separation Network steers a delay-based separator by inserting artificial channel delay. It admits a front-back ambiguity and sensitivity to microphone spacing. Spatial Aware Multi-Task Learning Based Speech Separation jointly separates and localizes, but it needs active ultrasonic sensing.
None of the three transcribes speech. None has a mechanism to stop trusting interaural cues when one channel is corrupted. BASNet's delay injection is close to our slow timing module, but it is set by hand to steer, whereas ours is estimated from data to correct.
Research Program
We will build a causal two-channel Conformer encoder with three input streams per frame: left and right log-spectra, interaural phase and level differences, and a monaural high-frequency spectral-shape stream per ear. The spectral-shape stream carries the pinna-filtering notches that let animals resolve front from back; Context-Specific Reweighting of Auditory Spatial Cues following Altered Experience during Development localizes reweighting to neurons with characteristic frequency above 8 kHz. A trust gate combines the interaural and monaural streams with weights that are recomputed every frame from short-window statistics, such as interaural coherence and per-ear level. This gate is the fast "context switch."
Before the interaural features are computed, a calibration layer applies a per-device fractional delay $\tau$ and per-band gain offsets $g_b$ to one channel. These are the slow, myelin-like parameters. They are frozen during training at zero and are updated only at deployment.
The decoder emits SOT sequences of the form ⟨time⟩⟨speaker⟩⟨azimuth bin⟩ words ⟨time⟩. The azimuth bins cover the full 360°, so front-back errors show up as distinct tokens.
Test-time recalibration updates only $\tau$ and $g_b$. It minimizes a label-free consistency loss over a sliding buffer of the model's own decoded output:
$$ L_{\text{cal}}(\tau, g) = \sum_t \big\| \hat\theta^{\text{ITD}}_t - \hat\theta^{\text{ILD}}_t \big\|^2 + \lambda \sum_{k} \sum_{t \in \text{turn}_k} \big\| \hat\theta_t - \hat\theta_{t-1} \big\|^2 . $$
Here $\hat\theta^{\text{ITD}}_t$ and $\hat\theta^{\text{ILD}}_t$ are azimuths predicted by auxiliary heads that read only timing cues or only level cues at frame $t$. The second term penalizes azimuth jumps within a decoded speaker turn $k$, and $\lambda$ balances the two terms.
The timing term identifies a delay offset. A delay offset moves the timing-based estimate but not the level-based one, so the two disagree. The level and monaural spectral cues identify gain offsets in the same way. The turn-smoothness term uses the diarization stream as the "feedback" that Behavioral training promotes multiple adaptive processes following acute hearing loss found necessary for adult remapping. Updates use a small step size and a long time constant, so momentary occlusions are left to the fast gate and only persistent offsets move the slow parameters.
Data come from LibriSpeechMix-style mixtures spatialized with head-related and binaural room impulse responses at RT60 from 0 to 0.7 s, following the protocol of the Wavesplit binaural benchmark, with moving-source trajectories. A held-out set uses real recordings from a head-and-torso simulator. Degradations are earplug-like unilateral high-frequency attenuation of 20 to 50 dB (the order of the threshold shift in Sound-Evoked Activity Influences Myelination of Brainstem Axons in the Trapezoid Body), interaural clock offsets of 50 to 500 µs, single-channel dropout, and abrupt on/off switching of each.
Baselines
The naive reference is a monaural speaker-attributed model in the style of End-to-End Speaker-Attributed ASR with Transformer. It is retrained on the better ear of our spatialized mixtures. Its reported result is 11.9% cpWER on monaural LibriCSS and 8.2% SA-WER (speaker-attributed word error rate) on LibriSpeechMix. It needs no calibration and cannot localize, so it sets the floor for what binaural input must add.
The strongest monaural joint decoder is SOMSRED (SOMSRED), with a reported 15.7% DER and 3.11% cpCER on 3-speaker simulated CSJ mixtures. We re-implement it on our English mixtures with the same encoder size, data, and tuning budget. Its CSJ numbers are context only, not a target on our data.
The strongest spatial competitor is a cascade. It runs the Wavesplit-style binaural separator with direction-of-arrival prediction (Online Binaural Speech Separation Of Moving Speakers With A Wavesplit Network, reported 8.2° direction error and 0.6 swaps per 24 s) into our SOMSRED re-implementation, with azimuth attached per separated stream. It is trained on identical data with matched total parameters and GPU-hours.
The most dangerous baseline is our own binaural SOT model trained with heavy multi-condition augmentation over all degradation types and with no test-time calibration. Every baseline that sees degraded audio gets the same augmentation budget.
Ablations isolate the key ingredient. The first removes the slow layer. The second removes the fast gate. The third merges them into one fast-adapting layer, which tests the "separate mechanisms" claim. The fourth replaces our consistency loss with entropy-minimization test-time adaptation on all encoder weights.
We count it as beating the baselines if, under persistent unseen-magnitude offsets, recalibration closes at least half of the gap in cpWER and azimuth error between the augmented model and the clean-condition model. It must do this without raising clean-condition error, and it must beat the cascade on speaker swaps and DER.
Why not just…?
Why not just train with heavy augmentation over channel degradations? This is the real threat. A network trained on enough corrupted pairs may learn implicit reweighting for free, which would make the fast gate redundant. The biological dual-map result hints at the cost: a single model trained on a mixture of conditions may settle on a compromise that is less precise when hearing is normal. That is an empirical question. If the augmented model matches ours on clean audio and on in-range offsets, we drop the gate claim. The proposal then rests on the slow recalibration of offsets outside the training range, such as a 400 µs clock skew when training covered 200 µs, which augmentation cannot cover by construction. Experiment 1 settles this directly.
Why not just estimate the channel offset with classical cross-correlation (GCC-PHAT) and subtract it? A fixed interaural delay offset shifts every source's apparent azimuth by the same amount. Correlation alone cannot tell "everyone moved left" from "the clock drifted." It needs an independent anchor. In our loss the anchor is the level and spectral cues and the turn structure from diarization. We will run GCC-PHAT plus median-offset compensation as a baseline. We predict it does well with a single static talker and fails with multiple moving talkers.
Why not just cascade a binaural separator into a monaural transcriber? Joint models have beaten cascades on monaural multi-talker speech, and the Wavesplit result shows separation alone still swaps speakers. The cascade is still a legitimate strongest competitor, and it is run with matched compute. Our specific advantage is that the cascade has no cross-task signal for recalibration, so its separator degrades under persistent channel offsets with no way to recover.
Why not just discard the corrupted ear and run monaurally? Better-ear selection is the monaural baseline. It cannot localize or resolve front from back. On clean audio, binaural input should help separate speakers with similar voices at different positions. If our model does not beat better-ear selection on cpWER under unilateral attenuation, the spatial pathway is not earning its cost.
Proposed Experiments
The first decisive check is a context-switching test. Unilateral attenuation is toggled on and off every 10 to 30 seconds within one conversation. We measure cpWER, DER, azimuth error, and front-back confusion rate during each regime. We also measure how many frames errors take to settle after each switch. The prediction from the dual-map account is that the gated model recovers within a few hundred milliseconds and shows no clean-regime penalty, while the merged-ablation and augmented models show either slow recovery or a clean-regime cost.
The second check is out-of-range remapping. Persistent clock offsets and gain asymmetries larger than any seen in training are applied for several minutes. We track $\tau$ and $g_b$ against ground truth, along with downstream cpWER and swaps. Success requires the estimated offset to converge to within one azimuth bin's equivalent delay, and transcription error to approach clean levels. Failure is equally informative: if consistency loss drifts or collapses onto degenerate azimuths, as entropy-based adaptation often does, that would show that cross-task agreement is not a sufficient feedback signal.
The third check is a negative control. On clean audio the slow parameters must stay near zero, and robustness on Speech Robust Bench's (Speech Robust Bench) monaural corruption suite, run on the averaged channel, must not degrade.
No reported result was found on Lacuna for binaural multi-talker transcription with azimuth tokens, so all binaural comparisons are against our matched re-implementations.
References
- Behavioral training promotes multiple adaptive processes following acute hearing loss
- Context-Specific Reweighting of Auditory Spatial Cues following Altered Experience during Development
- Sound-Evoked Activity Influences Myelination of Brainstem Axons in the Trapezoid Body
- SOMSRED: Sequential Output Modeling for Joint Multi-talker Overlapped Speech Recognition and Speaker Diarization
- Unified Multi-Talker ASR with and without Target-speaker Enrollment
- Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speech
- End-to-End Speaker-Attributed ASR with Transformer
- Online Binaural Speech Separation Of Moving Speakers With A Wavesplit Network
- Binaural Angular Separation Network
- Spatial-Aware Multi-Task Learning Based Speech Separation
- Speech Robust Bench: A Robustness Benchmark For Speech Recognition