Coupling Speaker Diarization and Multi-Talker Recognition Only at Their Outputs

When several people talk at once, a speech system has to answer two questions together: who is speaking at each moment (diarization) and what each person said (recognition). Recent systems answer both with one shared network sitting on top of a large pretrained speech encoder, and they report that training the two tasks jointly helps each one. The same papers also show the cost of this sharing. Training becomes unstable once a third speaker is added, and models trained on simulated mixtures that overlap all the time keep predicting overlap on real conversations, where people mostly take turns. This note asks where the joint-training benefit actually comes from. We suspect most of it comes from the two tasks agreeing on outputs: the same speaker ordering, and speech appearing in the same places. We suspect little of it comes from the tasks sharing internal features. If that is right, the benefit can be kept and the instability removed by giving each task its own small branch and connecting the branches only through their predictions.

Concretely, we freeze a self-supervised speech encoder. Such an encoder is pretrained on unlabeled audio by predicting hidden parts of the signal, for example WavLM or data2vec 2.0. On top of it we train two separate lightweight branches. One reads the lower layers, which carry speaker identity. The other reads the upper layers, which carry phonetic content. Neither branch shares any trainable parameters with the other. They interact only through a training loss that asks the recognizer's per-speaker "speech is happening here" signal to agree with the diarizer's per-speaker activity, under a single speaker ordering chosen jointly for both. At inference, the two activity estimates are combined at the decision level (late fusion). This combination also decides when to allow more than one active speaker at a time. The result is a multi-task system with no shared features, and a direct test of whether multi-talker synergy needs shared features at all.

Core Claim

In frozen-encoder multi-talker systems, the part where diarization and recognition interfere is the shared trainable module between the encoder and the task heads. The useful part of joint training can be carried by an output-level agreement loss alone. Our prediction is that output-coupled separate branches will match a shared-module system on simulated two-speaker mixtures. We further predict they will beat it on three-speaker mixtures and on real conversational audio, because a branch confused by overlap cannot corrupt the other branch's features. Two outcomes would refute the claim. The first is that coupled branches perform no better than branches trained independently and fused only at inference; that would mean output agreement transmits nothing. The second is that a shared module trained with careful loss balancing closes the gap; that would mean the problem was optimization, not sharing.

Where This Sits in Prior Work

Three systems motivate the question by showing synergy and also its fragility. Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder puts diarization, separation and recognition on one shared OWSMv3.1 encoder. It reports that joint training improves speaker representations, and also that three-speaker training collapses unless the recognition branch is pretrained separately. Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator inserts one small trainable separator (the "sidecar") into a frozen wav2vec 2.0 or data2vec 2.0 encoder. A 768-parameter diarization branch reads the sidecar's masks, and a shared speaker ordering is enforced across tasks. Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System applies the same sidecar idea to frozen Whisper. All three share features inside the network. We keep their frozen backbones and their shared-ordering idea, but we move the sharing from features to outputs.

A second group supplies techniques we reuse. End-to-End Neural Speaker Diarization with Permutation-Free Objectives introduced permutation-invariant training. This trick scores every possible assignment of predicted speaker tracks to true speakers and trains on the best one. The same paper also documents the simulation-to-real gap: 12.28% diarization error rate (DER) on simulated mixtures rises to 31.01% on CALLHOME, driven by over-predicted overlap. WavLM and Exploring wav2vec 2.0 on Speaker Verification and Language Identification show that speaker information sits in lower layers and content in upper layers. That finding justifies sending each branch to different layers. WavLM's overlap-simulating pretraining is also why we expect a frozen WavLM to be a usable multi-talker feature source.

A third group supplies the fusion design. Late Audio-Visual Fusion for in-the-Wild Speaker Diarization combines an audio diarizer and a visual diarizer at the level of per-speaker activity probabilities, using permutation-invariant score fusion that trusts the visual branch when it is confident. Work on adversarial robustness (Assessing the adversarial robustness of multimodal medical AI systems) argues that late fusion keeps corruption inside the stream where it starts. We borrow both ideas, but our two "streams" are two tasks on one audio signal rather than two sensors. We also couple the streams during training, which those papers do not.

One finding constrains the design. LibriMix reports that strong separation models do worse than simple masks when overlap is sparse. Any method that always assumes overlap will lose on real conversations. This is why our fusion step explicitly gates how many speakers may be active.

Research Program

The backbone is a frozen WavLM Base+ encoder, with data2vec 2.0 as a second backbone so we can compare directly with the sidecar paper.

The diarization branch takes a learned weighted sum of the lower and middle layers. It feeds this into a small convolutional separator and outputs $K$ per-frame speaker-activity probabilities $d_{k,t}$, where $k$ indexes output speaker tracks and $t$ indexes frames.

The recognition branch has its own separator on a weighted sum of the upper layers. It produces $K$ connectionist temporal classification (CTC) output streams. CTC is a loss that lets the model emit per-frame token probabilities, including a "blank" symbol, without needing alignments. A small speech-presence head on each stream gives $a_{k,t}$, the probability that stream $k$ is emitting speech at frame $t$. We use a dedicated head rather than one minus the CTC blank probability, because CTC outputs are spiky. The two branches share no trainable parameters. Their combined size is matched to the sidecar system's roughly 8.7M trainable parameters.

Training uses permutation-invariant CTC for recognition and binary cross-entropy for diarization. These are tied by one ordering $\pi$ chosen to minimize the sum of the two losses, plus an agreement term:

$$ \mathcal{L}_{\text{agree}} = \frac{1}{KT}\sum_{k=1}^{K}\sum_{t=1}^{T} \mathrm{BCE}\big(d_{k,t},\, a_{\pi(k),t}\big), $$

Here $T$ is the number of frames and BCE is binary cross-entropy. We test this loss in three gradient modes: flowing into both branches, flowing into diarization only, and flowing into recognition only. Comparing the modes tells us which direction carries the benefit.

At inference, the fused activity for each speaker is a calibrated log-linear combination of $d_{k,t}$ and $a_{\pi(k),t}$. The weights and temperatures are fit on a development set. On top of this sits an overlap gate. The fused probability that at least two speakers are active is thresholded, and below the threshold only the single most likely track may stay active. Words are assigned to speakers through the shared ordering $\pi$, which gives speaker-attributed transcripts with no separate attribution model.

The analysis has two parts. The first is a synergy decomposition: we train every combination of feature sharing (on or off) and output agreement (on or off), giving four systems from the same code. The second is a leakage probe. We corrupt only the input to one branch, through noise added at its layer weights or random masking of its separator, and measure how much the other task degrades. Late fusion predicts close to zero leakage. The shared sidecar predicts substantial leakage.

Baselines

The naive reference is a plain cascade. It uses the same frozen encoder, a diarization branch and a recognition branch trained completely independently, and word-to-speaker assignment by time overlap. This is late fusion with no training-time coupling, and it measures what separate branches give before any agreement loss is added. Next to it we run a fusion-only variant: the same independent branches, with the calibrated log-linear fusion and overlap gate applied at inference only. This is the ablation that isolates our key ingredient. If our system does not beat fusion-only, the agreement loss does nothing.

The strongest feature-sharing competitor is the sidecar system of Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator. It is the closest existing design: same frozen backbones, same parameter scale, and a shared ordering across tasks. It reports 14.62% DER (D2V-Sidecar-DB) and 14.89% DER (W2V-Sidecar-DB) on the two-speaker CALLHOME evaluation. The Whisper sidecar paper (Empowering Whisper) reports 9.69% word error rate (WER) for the D2V-Sidecar recognizer on Libri2Mix. We reimplement this system on our backbones with the same data, the same number of training steps and the same hyperparameter search budget, and we also check it against the published numbers. To beat it, we need lower CALLHOME DER than our reimplementation of the sidecar system, and Libri2Mix WER no worse than it, with a clear margin over seeds. We also need a larger margin on Libri3Mix, where we predict sharing becomes unstable.

As upper references with larger models, we report the published numbers for UME, which reaches 1.37% DER on Libri2Mix, 2.29% DER on Libri3Mix and 6.4% WER on clean mixtures (Unifying Diarization, Separation, and ASR). We also report Whisper-large with sidecar and target-talker identification, at 4.66% and 16.79% WER on Libri2Mix and Libri3Mix. These systems fine-tune or adapt much larger models, so we do not claim to beat them. The question is how much of their gap we close at about 9M trainable parameters.

For speaker-attributed transcription on LibriCSS, the reference is streaming t-SOT (Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings). It reports 12.0% concatenated minimum-permutation WER (cpWER), a WER computed after matching each speaker's words to the best reference speaker. For real-conversation diarization we also list EEND's 23.07% DER after CALLHOME adaptation. In the same paper, an x-vector clustering system scores 11.53% on CALLHOME, so clustering remains a strong reference on real telephone speech, and we report it honestly.

Why not just…?

Why not just run a cascade, diarizing first and then recognizing? The cascade is our naive baseline. If the agreement loss is useless, the cascade will match us. The UME paper argues that cascades accumulate errors and give up synergy. The fusion-only and cascade conditions test that argument directly, instead of assuming it.

Why not keep the shared module and fix interference with better loss weighting or gradient balancing? This is the most serious threat. If instability is just an optimization problem, a shared sidecar with tuned task weights could match us. We therefore give the shared baseline an extra search budget over task weights and over a schedule that pretrains the recognition branch first. That schedule is the fix UME needed for three speakers. If the tuned shared system matches ours on Libri3Mix and CALLHOME, we will report that feature sharing was not the problem.

Why not fine-tune one large shared encoder end to end, as UME does? This probably gives the best synthetic-mixture numbers; UME's 1.37% DER on Libri2Mix shows that. It also changes the question: it costs far more compute and erases the frozen-encoder setting. UME's own collapse at three speakers is evidence that more sharing makes stability worse. We compare against its published numbers and do not try to beat it on LibriMix.

Why not let each branch choose its own layers and stop there? Separate layer weights alone already give feature separation without coupling. That is exactly the cascade and fusion-only condition. Its role is to show whether separating features is enough on its own.

Proposed Experiments

We train on Libri2Mix and Libri3Mix, in both the clean and the noisy (WHAM!) versions. On the same mixtures we report DER and WER, plus the variance of each across five seeds as a measure of stability. Real-data transfer is measured with CALLHOME two-speaker DER, both zero-shot and after the same short adaptation EEND used, and with LibriCSS cpWER. For the sparse-overlap question we use SparseLibriMix, where we report the overlap false-alarm rate, meaning frames wrongly marked as having two speakers, separately for each overlap ratio from 0% to 100%. We expect the gate to remove most false alarms at low overlap without hurting recognition at high overlap. The file contains no reported SparseLibriMix numbers to compare against, so all systems there are our own runs. AliMeeting speaker-dependent character error rate tests whether the results hold in Mandarin meetings; CASA-ASR's 30.0% is a reference from a different and much heavier system.

Three results decide the claim. The first is the two-by-two synergy decomposition. Our claim needs the agreement-only system to recover at least most of the gain that the shared-sidecar system has over the cascade. The second is the three-speaker and CALLHOME comparison against the tuned shared baseline. Our claim needs output coupling to stay stable where sharing degrades. The third is the leakage probe. Our claim needs corruption in one branch to leave the other task close to its clean score, while the shared system degrades both.

If the first result fails, the paper becomes a negative result: synergy in multi-talker systems is a feature-level effect. That outcome is also worth knowing, because it tells designers that shared-feature instability has to be fixed rather than avoided.