When two people talk at once and a model must output one stream per talker, nothing in the training data says which output slot belongs to which talker. The standard fix, permutation invariant training (PIT), scores every way of pairing outputs with references and keeps the cheapest pairing. Recent self-supervised speech models have absorbed this trick to learn from overlapped audio. These are models pre-trained on unlabeled audio to predict masked parts of the signal. The trouble is what the cheapest pairing gets chosen from: per-speaker prediction of short acoustic units, which change every 20 milliseconds. As a result, nothing ties a slot to a person. Slots can swap talkers midway through an utterance, and a model run on consecutive chunks of a long recording has no way to know that slot 1 now is slot 2 a minute ago.
We propose to choose the pairing using the thing that is actually stable in speech, the voice. Each output slot is split into a slow code, one vector per utterance that should describe who is speaking, and a fast stream that should describe what is said. The slot pairing is decided once, by matching voice codes, and the fast stream is then conditioned on that voice code. The model therefore extracts each talker using a speaker key it discovered itself, with no enrollment recording. That is the property speaker-conditioned models lack, and the property monolithic PIT models never produce. Because voice codes are comparable across chunks and across mixtures, the same mechanism stitches long recordings into consistent tracks. Disentangling speaker from content is not a side product here. It is the means of making permutation training stable and long-form consistent.
Core Claim
Moving the permutation decision out of the frame-level unit space and into a learned, time-invariant speaker space should give three things in a single pre-trained model. First, it should yield fewer mid-utterance track swaps than frame-level PIT. Second, it should give consistent speaker tracks across chunks of long recordings, through simple matching of voice codes. Third, it should approach the multi-talker accuracy of models that require an enrolled reference for each speaker, without needing one. The new element is the source of the assignment: a self-learned voice code, trained only from the knowledge of which simulated source went into which mixture. It does not come from content tokens as in prior PIT, from an external enrollment, or from arrival order.
Where This Sits
Three works define the gap. Cocktail HuBERT gives a HuBERT backbone $K$ prediction heads and applies PIT over each source's k-means unit targets. It reports 7.8% WER on Libri2Mix multi-talker ASR at LARGE scale. Its heads are monolithic, however: nothing makes a head carry a stable identity, so it supplies our training recipe but not our assignment mechanism. SA-WavLM avoids permutations entirely by conditioning extraction on an enrolled speaker embedding through conditional layer normalization, a layer whose scale and shift are set by the speaker vector. It reaches 8.39% WER on Libri2Mix and 1.88 DER on SUPERB diarization. It shows that a speaker key is a powerful extraction cue, but it needs clean enrollment audio. We keep its conditioning design and replace the enrollment with a self-discovered key. SpeechTokenizer separates content from residual acoustics in a single-speaker codec by distilling HuBERT into its first quantizer layer. It motivates the content/voice split but breaks on overlapped input.
Other work constrains the design. AudioSlots uses Hungarian-matched latent slots but processes 0.5-second chunks independently and has no track continuity. That is exactly the failure that cross-chunk voice-code matching targets. Group-PIT handles long-form meetings by pooling utterances into a fixed number of channels, but it still resolves assignment by signal loss. Stabilizing Label Assignment and interrupted and cascaded PIT document early-training label flipping and fix it through initialization or frozen assignments. We fix it by making the assignment depend on a quantity that varies slowly by construction.
Vocal harmony separation is a direct warning: when sources share timbre, identity cues are weak. Our method should be weakest for similar-sounding talkers, and we will measure this rather than hide it.
Research Program
The backbone is a HuBERT-BASE-sized Transformer pre-trained on simulated LibriSpeech mixtures with one to three talkers and noise, following the Cocktail HuBERT simulation. On top of it sit $K$ slots. Each slot $k$ produces a voice code $v_k$, obtained by attention-pooling over the whole utterance so that it cannot carry frame-level content. It also produces a content stream that is passed through a narrow bottleneck. The bottleneck dimension follows the AutoVC logic that a narrow, downsampled channel cannot hold global voice texture. The content stream is modulated by $v_k$ and trained to predict the masked k-means units of source $k$, computed from the clean source.
The assignment is computed once per utterance from voice codes alone. Let $u_s$ be a reference voice code for source $s$, obtained by running the same pooling module on the clean source through a momentum copy of the network. The pairing is
$$\pi^* = \arg\min_{\pi} \sum_{k=1}^{K} \big(1 - \cos(v_{\pi(k)}, u_k)\big),$$
which the Hungarian algorithm solves in $O(K^3)$, as in Dovrat et al.. The masked unit cross-entropy for every slot is then computed under $\pi^*$, so content gradients can never select a different pairing from the voice pairing.
The voice codes become identity codes without speaker labels through a second ingredient. The same source utterance is placed into two different mixtures, with different partners, crops and noise. A contrastive loss pulls together the two voice codes for that source and pushes apart the codes for other sources in the batch. Unlike speaker-verification pre-training such as Jia et al., this needs no speaker labels, only knowledge of which source was mixed where, which simulation gives for free. A risk is that $v_k$ might encode utterance-specific content instead of voice. A variant pairs two different utterances of the same LibriSpeech speaker, using LibriSpeech's speaker metadata, to test whether stronger identity supervision helps.
For long recordings, the model runs on overlapping chunks. Tracks are linked by Hungarian matching of voice codes between adjacent chunks, plus a running per-track average code. No separate clustering stage is used.
Baselines
The naive reference is vanilla HuBERT BASE and WavLM BASE fine-tuned on each downstream task through the SUPERB protocol. SA-WavLM reports WavLM Base at 13.30% WER on Libri2Mix and 4.55 DER on SUPERB diarization, so a model that does not clearly beat these has learned nothing useful from mixtures.
The strongest blind competitor is Cocktail HuBERT. The published 7.8% Libri2Mix WER is for the LARGE model on 60,000 hours. The reported BASE figure we can compare against is 3.86% DER on Libri(2+3)Mix, plus 2.77 DER on SUPERB diarization as cited by SA-WavLM. Because we pre-train at BASE scale, we will retrain C-HuBERT BASE ourselves on identical simulated mixtures, with the same number of updates, the same masking and the same k-means targets. Our only change is how the pairing is chosen plus the voice/content split. Beating it means lower Libri2Mix WER and DER at matched compute, together with a substantially lower track-swap rate. Matching it on WER while cutting swaps and fixing long-form consistency would still be a meaningful result.
SA-WavLM serves as an upper reference in two forms, both retrained on the same data. With oracle enrollment from clean held-out speech, it marks what a perfect speaker key buys: 8.39% WER and 1.88 SUPERB DER reported. With pseudo-enrollment, embeddings are taken from single-talker regions found by a diarizer, which is the realistic blind deployment. Closing most of the gap to the oracle version without enrollment, and beating the pseudo-enrolled version, is the success criterion.
A cascade baseline runs a strong supervised separator, DPTNet as in Huang et al. at 21.3 dB SI-SNRi on WSJ0-2mix, followed by HuBERT BASE on each stream. An arrival-order baseline replaces PIT with a Sortformer-style sort loss, ordering slots by speaker onset. The key ablation keeps every component of our model but chooses the pairing by minimum unit loss per utterance, which is C-HuBERT-style PIT on top of the split architecture. This isolates the assignment rule from the extra capacity of the voice and content heads.
Why not just…?
Why not just separate first and run a single-speaker model on each output? On fully overlapped synthetic mixtures, a cascade may well match us on WER; no reported result found on Lacuna for a separator-plus-HuBERT cascade on Libri2Mix ASR, so we run it. The cascade has three costs: two models, separator artifacts in a domain HuBERT never saw, and no answer to cross-chunk track identity, because the separator's output order is itself permutation-arbitrary per chunk. LibriMix also reports that Conv-TasNet-style separators do worse than simple masks at low overlap, which matters for real conversation. If the cascade wins on long-form swap rate as well, our claim fails.
Why not just use C-HuBERT and cluster its head outputs afterward to link chunks? This is the most plausible competitor for the long-form question. Our answer is empirical: C-HuBERT heads are trained only to predict content units, so pooled head features have no pressure to be speaker-consistent across mixtures. We will measure speaker-verification equal error rate (EER) from pooled C-HuBERT head features against our voice codes, and compare chunk-stitched DER for both. If post-hoc pooling performs as well, the voice-code training is unnecessary and we will say so.
Why not just sort speakers by arrival time? Sort losses remove the ambiguity cheaply, but they define order by onset. That is ill-posed when talkers start together, and it says nothing about which track in the next chunk is the same person. The arrival-order baseline, evaluated on simultaneous-onset subsets and on stitched long-form sessions, settles this directly.
Why not just obtain enrollment from a diarizer and use SA-WavLM? This is a real threat when single-talker regions are plentiful. The pseudo-enrolled SA-WavLM baseline tests it, and we report results by overlap ratio using SparseLibriMix. We expect the advantage of self-enrollment to appear at high overlap, where clean enrollment snippets are scarce.
Proposed Experiments
Pre-training uses simulated mixtures from LibriSpeech 960 h with DNS-style noise, at BASE scale for all retrained systems. Downstream evaluation covers Libri2Mix multi-talker ASR WER, Libri(2+3)Mix and SUPERB diarization DER, Libri2Mix separation SI-SDRi in the 1% low-resource setting (SA-WavLM reports 8.42 dB against 3.97 dB for HuBERT Base), and the single-speaker SUPERB tasks. The single-speaker tasks check the "performance tax" that C-HuBERT reported at BASE scale.
Three checks decide the outcome. The first is the track-swap rate: the fraction of utterances whose per-frame best pairing changes within the utterance, measured on Libri2Mix and Libri3Mix and binned by the cosine similarity of the two talkers' reference embeddings. Our method should cut swaps most where voices differ, and we report honestly how it behaves for similar-sounding pairs, the regime the vocal-harmony results warn about. The second is a long-form test: minute-long sessions concatenated from LibriSpeech utterances with SparseLibriMix-style overlap ratios from 0% to 100%, scored with chunk-stitched DER and speaker-attributed WER against C-HuBERT plus post-hoc clustering and the arrival-order model. The third is a set of disentanglement probes. Speaker-verification EER from the content stream should be near chance, and unit-prediction accuracy from the voice code should be near chance; together they confirm that the assignment really rests on voice. Training-curve swap counts over the first epochs, compared against frame-loss PIT, test the stability claim from Huang et al. without any supervised warm-up.
References
- Cocktail Hubert: Generalized Self-Supervised Pre-Training for Mixture and Single-Source Speech
- SA-WavLM: Speaker-Aware Self-Supervised Pre-training for Mixture Speech
- SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models
- Audioslots: A Slot-Centric Generative Model For Audio Separation
- Separating Long-Form Speech with Group-wise Permutation Invariant Training
- Stabilizing Label Assignment for Speech Separation by Self-Supervised Pre-Training
- Interrupted and Cascaded Permutation Invariant Training for Speech Separation
- Vocal Harmony Separation Using Time-Domain Neural Networks
- AutoVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
- Many-Speakers Single Channel Speech Separation with Optimal Permutation Training
- Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
- Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems
- LibriMix: An Open-Source Dataset for Generalizable Speech Separation