Speech recognizers break when two people talk at once, and the best fixes are trained almost entirely on artificial overlap: clean audiobook sentences from two strangers added on top of each other. Real conversation does not look like that. Overlaps are short and sparse, speakers know each other and take turns, and everything carries the echo of one particular room. The audio that has these properties sits on meeting-room devices, smart speakers and phones. It is unlabeled and private, so it cannot be collected. This note proposes federated self-mixing. Each device finds stretches of its own audio where only one person is talking. It transcribes them with an ordinary single-speaker recognizer, which is reliable on clean speech. It then mixes its own segments together at the overlap rates it actually observes, which yields labeled overlapped training data in its own room, with its own speakers, and with no human transcription. Devices train only a small overlap-handling module on these mixtures and send that module's weights, never audio, to a server that averages them. This is federated learning: the model travels to the data instead of the data to the model.
In two sentences: the hard part of learning overlapped speech from private devices is that nobody can label overlap there, but a device can label clean single-speaker speech and build its own overlap from it. Averaging small modules trained this way across many rooms should teach a recognizer what natural overlap sounds like, which simulation from public corpora does not.
Core Claim
A multi-talker recognizer first trained on public simulated mixtures, then adapted by federated averaging of a lightweight overlap module on locally self-mixed, self-labeled audio, will recognize real conversational overlap more accurately than three alternatives: the same model trained centrally on simulated mixtures, even when that simulation is matched to natural overlap statistics; the same module trained on one device alone; and federated self-training that pseudo-labels the real overlap directly. It will do this without hurting single-speaker accuracy, and without sending speaker identity off the device.
Where This Sits
Overlapped speech recognition has converged on serialized output training, or SOT: transcripts of several speakers are written as one token stream with speaker-change markers, so a standard decoder can emit any number of speakers (Serialized Output Training for End-to-End Overlapped Speech Recognition). Two refinements feed speaker information into the decoder to stop it blending words across speakers. SA-SOT fuses speaker embeddings and reweights attention by speaker similarity, reaching 3.41% concatenated minimum-permutation WER (cpWER) on LibrispeechMix. SC-SOT conditions the decoder on a jointly trained diarizer, the component that decides who speaks when. Both methods supply our model components, and both are trained and tested only on simulated, fully overlapped LibriSpeech mixtures. Cascaded encoders for fine-tuning ASR models on overlapped speech supplies our architecture. It keeps a large pretrained single-speaker model frozen, adds a small "mask encoder" for overlap, and uses a linear probe that labels each frame as overlapped or not with 91% accuracy. That design, which cut overlapped WER by 9% at parity on clean speech, is exactly the shape a bandwidth-limited federated system needs.
Three findings motivate the gap. LibriMix shows that separation models trained on full overlap do worse than simple masks when overlap is sparse (0–20%), which is the regime of real talk. Evaluating the Santa Barbara Corpus and Evaluating Open-Source ASR Systems show that large pretrained models fail on spontaneous conversational overlap. The data that would fix this is in-the-wild conversation, which is the data federated learning exists to reach.
Federated speech work so far assumes one speaker per client. Federated Representation Learning for ASR pretrains an encoder with self-supervision on speaker-siloed Libri-Light and fine-tunes the decoder centrally. We instead federate the overlap-specific supervised stage on multi-speaker clients. Parameter-Efficient Transfer Learning under Federated Learning for ASR shows that averaging only small adapter layers costs 4–8% of full-model communication and matches centralized tuning, but it assumes labeled single-speaker client data. We reuse its communication argument and remove the label requirement. DecoupleFL handles unlabeled clients with teacher–student pseudo-labels and protects privacy by erasing speaker identity from shared features. That erasure would destroy exactly the speaker cues SA-SOT and SC-SOT depend on, so DecoupleFL constrains our design: we keep speaker information but keep it local. FedFSS shows that federated speaker embeddings collide across clients unless they are anchored. That result is a second reason not to average a speaker branch globally.
Research Program
The model is a frozen pretrained single-speaker Conformer encoder and decoder. On top of it sits a cascaded mask encoder with SOT output, following Rose et al., and an overlap-detection probe. The mask encoder is warm-started centrally on Libri2Mix/Libri3Mix-style simulated mixtures. A small speaker-fusion block in the style of SA-SOT's embedding fusion conditions the mask encoder on embeddings of the speakers present.
Each federated round, a client does four things. First, the overlap probe and a voice-activity detector split the client's recordings into single-speaker regions, overlapped regions and silence, and the client records its empirical overlap-length and turn-gap distributions. Second, the frozen single-speaker recognizer transcribes the single-speaker regions, and low-confidence segments are discarded. Third, the client builds mixtures from pairs and triples of its own segments. Overlap ratios and offsets are sampled from its own statistics, and the SOT targets are assembled from the pseudo-transcripts. Because the segments were recorded in the same room by the same device, the mixtures inherit the real reverberation, microphones and speaker population. Fourth, the client trains the mask encoder on these mixtures with the SOT loss and a short replay of its clean segments to protect single-speaker accuracy. It sends only the mask-encoder update; the server averages updates in proportion to the number of mixtures each client built.
The speaker-fusion block and a per-client table of speaker embeddings never leave the device. They are personal and deliberately unaveraged, consistent with the mapping conflicts FedFSS documents. The server never sees a speaker vector.
Two analyses accompany the system. The first measures how pseudo-label error on clean segments, and contamination from the 91%-accurate overlap probe, propagate into cpWER as the confidence threshold varies. The second characterizes how far self-mixed audio departs from natural overlap. Summing two distant-mic recordings doubles background noise and drops the reactive timing of real interruptions, and we quantify the cost of both by comparing against the small set of truly overlapped regions with references.
Baselines
The naive reference is the warm-started model with no adaptation: the cascaded SOT model trained only on public simulated mixtures. It shows how much natural overlap is missed when nothing is learned from the target conversations.
The strongest existing method is a centrally trained speaker-conditioned SOT model in the SC-SOT style, trained with diarization multi-task learning on the same public simulated data. SC-SOT reports 17.5% / 15.2% WER on Libri2Mix dev/test, against 19.2% / 16.8% for plain SOT, and 33.2% / 30.9% on Libri3Mix. SA-SOT reports 3.41% cpWER on LibrispeechMix. We will reproduce our backbone's numbers on Libri2Mix so these serve as sanity anchors. For real meeting or conversational audio, no reported result was found on Lacuna for these methods. The Volcspeech system for the ICASSP 2022 multi-party meeting challenge applies SOT to such data, but the file gives no number, so that baseline must be run ourselves.
The fairest and most threatening baseline is centralized matched simulation. It builds mixtures from public single-speaker corpora using the pooled overlap statistics measured on clients, and it adds room-impulse and noise augmentation. It runs with the same architecture, the same number of training mixtures, the same optimizer steps and the same tuning budget. Federated self-mixing beats it only if the remaining gain comes from in-room acoustics and real speaker populations rather than from the overlap ratio alone.
Two federated baselines complete the set. Local-only training applies identical self-mixing on each client with no averaging. Federated real-overlap pseudo-labeling has the warm-started model transcribe each client's natural overlapped regions and trains on its own SOT outputs with an exponential-moving-average teacher, in the manner of DecoupleFL's teacher–student scheme. Both use the same rounds, client sampling and communication budget.
Two ablations isolate the key ingredients. The first replaces each client's own segments with public LibriSpeech segments convolved into the client's room. This separates the contribution of the client's speakers from that of the room. The second averages the speaker-fusion block globally instead of keeping it local. "Beating" a baseline means a statistically reliable cpWER reduction, using bootstrap over sessions, on held-out real overlapped regions, with no more than a 2% relative increase in single-speaker WER.
Why not just…?
Why not just simulate better overlap centrally from public speech? This is the real threat. If matched simulation with room augmentation recovers most of the gain, federation is unnecessary, and we will report that. The LibriMix sparse-overlap failure and the Santa Barbara results suggest simulated statistics alone are not enough, but they do not show that in-domain acoustics and speakers are what is missing. The matched-simulation baseline and the public-segments-in-client-rooms ablation are designed to settle this directly.
Why not just fine-tune on each device separately? A meeting room or household has a handful of speakers and limited hours. A module tuned there will probably overfit its few voices and fail on new visitors. The local-only baseline, evaluated on held-out sessions with unseen speakers, measures this. Federation is justified only if averaging beats it on unseen speakers while staying close to it on familiar ones.
Why not just pseudo-label the real overlapped audio with the current multi-talker model? Because that model is least reliable exactly where it is needed. Centralized SOT models still sit at roughly 15–31% WER on two- and three-speaker mixtures (SC-SOT), and self-training on such labels tends to reinforce speaker confusions. Self-mixing instead draws labels from clean speech. The federated real-overlap pseudo-labeling baseline tests this; if it wins, the self-mixing premise is wrong. A combination of the two is a natural follow-up.
Why not just separate the speakers first and run a federated single-speaker recognizer on each stream? Cascades carry separation artifacts and speaker-channel swaps into recognition. LibriMix shows that separators trained on full overlap distort the sparse-overlap regime. We include a separation-plus-single-speaker cascade, tuned on the same client data, as a secondary comparison.
Proposed Experiments
Clients are simulated from real multi-party recordings by treating each meeting session or room as one client. The multi-channel meeting data of the ICASSP 2022 meeting transcription challenge is used single-channel, and the Santa Barbara Corpus of conversational English serves as an out-of-domain conversational test. Client training uses audio only. Reference transcripts are used solely to score held-out sessions. Public warm-start data comes from LibriMix and LibrispeechMix, and Libri2Mix and Libri3Mix test sets are re-scored after adaptation to check for regression on simulated overlap.
Metrics are cpWER on segmented overlap regions, and MIMO WER for long unsegmented sessions. MIMO WER is the polynomial-time approximation that penalizes speaker-channel switches, introduced because exact multi-speaker assignment is NP-hard (On Word Error Rate Definitions and Their Efficient Computation for Multi-Speaker Speech Recognition Systems). We also report single-speaker WER, bytes uploaded per client per round, and results stratified by overlap ratio and by number of simultaneous speakers.
The decisive check is a single comparison on held-out real overlap: federated self-mixing against centralized matched simulation at equal compute and data volume. A reliable gain, which we would consider meaningful at around 10% relative cpWER, together with a gain over federated real-overlap pseudo-labeling, confirms the claim. A gain that vanishes in the public-segments-in-client-rooms ablation would show that rooms, not speakers, carry the benefit. No gain over matched simulation would mean natural overlap can be simulated well enough, and federation is not worth its cost for this task.
References
- Serialized Output Training for End-to-End Overlapped Speech Recognition
- SA-SOT: Speaker-Aware Serialized Output Training for Multi-Talker ASR
- SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition
- Cascaded encoders for fine-tuning ASR models on overlapped speech
- LibriMix: An Open-Source Dataset for Generalizable Speech Separation
- Evaluating the Santa Barbara Corpus: Challenges of the Breadth of Conversational Spoken Language
- Evaluating Open-Source ASR Systems: Performance Across Diverse Audio Conditions and Error Correction Methods
- Federated Representation Learning for Automatic Speech Recognition
- Parameter-Efficient Transfer Learning under Federated Learning for Automatic Speech Recognition
- Decoupled Federated Learning for ASR with Non-IID Data
- Federated Learning with Feature Space Separation for Speaker Recognition
- On Word Error Rate Definitions and Their Efficient Computation for Multi-Speaker Speech Recognition Systems