Speech recognizers that transcribe several people talking at once work best when they already know what each speaker sounds like. In practice they rarely do. Nobody records a voice sample before joining a meeting, so the system meets every speaker for the first time. This is a cold-start problem, the same one recommender systems face with a brand-new user who has no history. The strongest current answer in speech runs a separate diarizer (a model that detects who is speaking when) and then launches one recognizer copy per detected speaker. That works, but the cost grows with the number of people in the room.
We propose a different route. The model enrolls speakers itself, inside a single decoding pass. We build on serialized output training (SOT), which turns overlapped speech into one token stream with a special token marking each speaker change and decodes it with a single decoder. We make those change tokens refer to a small speaker memory that the model writes while it decodes. The first time the decoder emits "speaker 3", it pools the audio frames it attended to while producing that speaker's words into memory slot 3. Every later segment it attributes to speaker 3 refines the slot. Later decoding reads the slots, so a speaker who returns after a long silence is recognized against a voiceprint the system built from its own earlier output. The claim is that this in-pass self-enrollment closes most of the gap between cold and pre-enrolled multi-talker recognition, at roughly single-decoder cost, and that the gain grows with speaker count and session length. Those are exactly the conditions where plain SOT degrades.
Core Claim
A single-branch SOT decoder augmented with a bounded, self-written speaker memory will match or approach parallel-instance self-speaker adaptation on 3-speaker mixtures and long meetings. It will do so without an external diarizer and with compute that grows with the number of memory slots rather than the number of decoder copies. The gains should come mainly from speakers who re-enter after gaps, because that is where a memory helps and a local attention window does not.
This is a claim about recognition accuracy, not only about labeling who said what. If the memory improved speaker labels but left word errors unchanged, a post-hoc clustering step would do the same job more simply. We treat that outcome as a falsification.
Where This Sits
The SOT line supplies the core mechanism. Serialized Output Training for End-to-End Overlapped Speech Recognition concatenates transcripts in order of start time with a speaker-change token. It showed that one decoder can count speakers and avoid duplicated words. Streaming Multi-Talker ASR with Token-Level Serialized Output Training interleaves words by time across "virtual channels" for streaming. t-SOT FNT switches the internal language-model state per channel so each thread stays grammatical. None of these three keeps any persistent notion of speaker identity: the change tokens mark a switch but say nothing about who is switching in. We add that identity through the memory.
Two papers add speaker information and constrain our design. SA-SOT fuses embeddings from a dedicated speaker branch and reweights decoder self-attention with a token-by-token speaker similarity matrix. It reaches 3.41% cpWER on LibriSpeechMix, but the full matrix is costly, and it assumes clean speaker embeddings rather than cold ones. We replace the quadratic token-by-token matrix with attention over a handful of slots. SOT Triggered Neural Clustering for Speaker Attributed ASR uses SOT's speaker count to trigger a separate clustering decoder over pre-extracted speaker embeddings on AMI. There, attribution happens after recognition and cannot feed back into it. Ours runs inside decoding.
The paper that motivates this work most directly is Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR. It solves the speaker cold start with an online diarizer whose activity masks gate a learnable speaker kernel. It then runs one recognizer instance per speaker and reports 5.0% cpWER on 3-speaker LibriSpeechMix, against 24.0% for SOT. The authors name scaling to larger groups as an open limitation. That gap between SOT and SSA on 3-speaker mixtures is the target of this proposal.
From the recommendation cold-start literature we borrow one idea and reject another. MAMO showed that a single global initialization fails when new users differ from the average. Its remedy is a memory that yields a personalized bias from few interactions: 0.8894 MAE against MeLU's 0.9299 on MovieLens-1M cold-user/cold-item, within 5–15 interactions. Our slots play that role for speakers. MAMO and Customized Conversational Recommender Systems both adapt through per-user gradient steps, and both note the heavy training cost of that two-level optimization. We write the memory in a forward pass instead, with no test-time gradient steps. NextQuill concedes that its preference attribution fails when a user has no history. That supports our premise: identity signals must be bootstrapped from the session itself.
Research Program
The model starts from an attention-based SOT recognizer. We replace the generic speaker-change token with relative speaker tokens for slots 1 through K. These are assigned in order of first appearance, which keeps SOT's first-in-first-out training target and its linear-cost labeling. With K around 8, one slot per possible speaker, the tokens need no enrollment.
The memory is written in two ways. When the decoder emits a slot token not yet used in the session, a write step initializes that slot. It averages encoder frames, weighted by the decoder's cross-attention over the tokens of the segment that follows. When the decoder emits a slot token already in use, the same pooled vector updates the slot through a learned gate.
The memory is read in two ways. Every decoder step cross-attends to the K slots in addition to the audio. The prediction of which slot comes next is scored partly by the similarity between each slot and the upcoming audio. This lets the model choose between opening a new slot and returning to an old one.
Training uses teacher forcing: slot contents are written from the ground-truth speaker order. We add a small contrastive loss that keeps slots of different speakers apart. To reduce the mismatch between training and inference, we gradually train on memories written from the model's own predicted speaker labels.
For long-form audio, the recognizer processes fixed windows and carries the memory across windows while discarding the decoder's token history. The memory is the only cross-window state. That design isolates whether persistent identity, rather than longer text context, drives any improvement.
The analysis follows from the claim. We break down errors by whether a segment belongs to a speaker's first appearance or a return, by the length of the gap before the return, and by speaker count. We also measure how much of the cold-to-enrolled gap is closed, defined as the cold baseline's cpWER minus ours, divided by the cold baseline's cpWER minus that of an oracle-enrolled system.
Baselines
The naive reference is plain sentence-level SOT with generic change tokens, trained on the same data and architecture. On 3-speaker LibriSpeechMix, SSA's paper reports it at 24.0% cpWER. Our own reproduction serves as the working reference so that only the memory differs.
The strongest competitor is SSA from Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR. It reports 5.0% cpWER on 3-speaker LibriSpeechMix and 5.9% on 2-speaker streaming at 560 ms latency. It is the strongest because it solves the same cold-speaker problem with the best reported number. We compare in two ways. Against its published figure, beating it means reaching at most 5.0% on 3-speaker LibriSpeechMix. We also rerun it with an encoder of matched size and the same training data, and report decoding compute per audio second. We count it as a win if we match SSA within about half a point of cpWER while using clearly less compute at three or more speakers, or if we beat it outright on long-form sets.
The strongest SOT-side competitor is SA-SOT, with its reported 3.41% cpWER on LibriSpeechMix and 12.75–22.03% relative gains over CIF t-SOT on the 2-mix sets. We run it with the same tuning budget and measure both accuracy and attention cost as session length grows.
For attribution on AMI, we compare against SOT-triggered clustering. The file reports only relative gains for it: 19% relative DER reduction, and 7% and 4% relative cpWER improvement on the development and evaluation sets. No absolute AMI number was found on Lacuna, so we reproduce it.
Three ablations isolate the key ingredient. The first reads the memory but never updates it after first appearance. The second keeps the slot tokens but removes the memory, so identity is only a label. The third replaces self-written slots with slots from an external pretrained speaker encoder applied to the same segments. An oracle-enrolled system that fills the slots from true speaker audio gives the upper bound.
Why not just…?
Why not just run SOT and cluster speaker embeddings afterward? This is the most serious threat. SOT Triggered Neural Clustering already improves attribution this way. If post-hoc clustering matches our speaker attribution and our word error rate, the memory adds nothing. Our distinguishing prediction is that the memory lowers word errors, especially on returning speakers, because it guides decoding rather than labeling afterward. The no-memory ablation plus clustering, compared against the full model on the returning-speaker slice, settles this directly.
Why not just use SSA with parallel instances? It is more accurate than any published single-branch system at three speakers, per Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR. Its cost, though, grows with the number of decoder copies, and it depends on an external diarizer whose errors pass straight into recognition. We may not beat it on accuracy on short mixtures. The proposal succeeds only if we give up little accuracy for a large compute reduction, or win on long meetings where the diarizer struggles. Reporting accuracy against compute at two, three, and more speakers makes this trade-off visible rather than hiding it.
Why not just adapt to each new speaker with a few gradient steps, as meta-learning does for new users? MAMO and Customized Conversational Recommender Systems show that this works from a handful of examples. Both also document the heavy cost of the two-level optimization. At test time it would need backpropagation in the middle of the stream, which conflicts with streaming latency. We include a test-time-adaptation variant, in which the slot vectors are updated by a few gradient steps on the model's own pseudo-labels, to check whether the forward-pass write loses accuracy.
Why not just give SA-SOT longer context? Its token-by-token similarity matrix grows quadratically with length, and more text context does not keep a stable voice identity. Our cross-window experiment, in which only the memory persists, tests whether identity rather than length is what helps.
Proposed Experiments
We train on simulated LibriSpeech mixtures with 1–4 speakers. We evaluate on LibriSpeechMix 1-, 2-, and 3-speaker sets using cpWER, and on LibriCSS across overlap ratios, where t-SOT reports 9.0% average WER. For real conversations we use AMI with cpWER and DER, and CH109, where no absolute number was found on Lacuna for SSA.
For long form, we build LibriSpeech-derived sessions in which each speaker takes several turns separated by controlled gaps. This lets us plot error against gap length and against speaker count up to six.
The decisive checks are three. First, on 3-speaker LibriSpeechMix, the full model must cut SOT's error substantially toward SSA's 5.0% and close most of the cold-to-oracle gap. Second, the frozen-memory and no-memory ablations must lose most of that gain, concentrated on returning speakers. Third, the self-written memory must match or beat slots filled by an external speaker encoder. If it does not, the value lies in the memory structure rather than self-enrollment, and the claim narrows accordingly.
References
- Serialized Output Training for End-to-End Overlapped Speech Recognition
- Streaming Multi-Talker ASR with Token-Level Serialized Output Training
- T-SOT FNT: Streaming Multi-Talker ASR with Text-Only Domain Adaptation Capability
- SA-SOT: Speaker-Aware Serialized Output Training for Multi-Talker ASR
- SOT Triggered Neural Clustering for Speaker Attributed ASR
- Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR
- MAMO: Memory-Augmented Meta-Optimization for Cold-start Recommendation
- Customized Conversational Recommender Systems
- NextQuill: Causal Preference Modeling for Enhancing LLM Personalization