Speaker recognition systems learn to separate who is speaking from how the audio got to us: the microphone, the room, the phone codec, the background noise. The classical tool for this separation is factor analysis. It writes each recording's representation as a speaker part plus a channel part plus noise, and it scores identity after setting the channel part aside. Every version of this tool assumes it can see the whole recording, or a labelled collection of recordings, before it estimates the channel. A live stream offers neither. In a meeting feed, a call-center line, or a broadcast, the channel changes partway through, the speaker changes many times, and decisions must be made within a fraction of a second, without future frames and without labels.
We propose to model the channel as a slowly drifting hidden state that persists across speaker changes, and the speaker as a quantity that jumps at turn boundaries and recurs over time. A filter that runs forward in time can then separate the two causally, using one simple cue: when different speakers alternate under the same channel, the shift they share is channel, and when the channel moves, every speaker moves with it. In effect, this gives a principled replacement for the utterance-level normalization that streaming systems have had to drop, and it removes the need for the nuisance labels that current multi-condition factor models require. We pair it with a strictly causal evaluation protocol, so that the comparison with offline systems measures real deployment behaviour and not hindsight.
Core Claim
Joint Probabilistic Linear Discriminant Analysis (JPLDA) shares a nuisance factor across all recordings that carry the same condition label. A PLDA model is a linear-Gaussian model of speaker embeddings with separate between-speaker and within-speaker covariance. JPLDA is described in A Generalization of PLDA for Joint Modeling of Speaker Identity and Multiple Nuisance Conditions, which reports 5% to 47% relative minimum detection cost improvement over standard PLDA. Our claim is that in a stream, temporal adjacency can stand in for the condition label. A nuisance factor tied across neighbouring windows by a random-walk prior keeps most of JPLDA's benefit. Because this tying is causal and needs no labels, it also handles mid-stream channel changes that a static, label-tied model cannot represent. For each window $t$ of the stream, with embedding $e_t$ extracted from a short causal window, we write
$$e_t = \mu + F h_{s_t} + G y_t + \epsilon_t, \qquad y_t = y_{t-1} + w_t,$$
where $\mu$ is the global mean, $F$ and $G$ are speaker and channel loading matrices learned offline, $h_{s_t}$ is the latent identity of the speaker $s_t$ active in window $t$, $y_t$ is the low-dimensional channel state, $w_t \sim \mathcal{N}(0, Q)$ is its per-window drift with covariance $Q$, and $\epsilon_t$ is residual noise.
Suppose the speaker assignments $s_{1:t}$ are fixed. Everything is then linear-Gaussian, so the joint posterior over the channel state and the identities of the speakers seen so far updates exactly in closed form, as in a Kalman filter. Only the discrete assignment is hard. We handle it with a small set of weighted assignment hypotheses, which is a Rao-Blackwellized particle filter: particles carry the discrete choice, and closed-form updates handle the continuous parts.
Research Program
The first component is the filter itself. We would build it on embeddings from a CNN trained on in-the-wild audio, the model family that Voxceleb: Large-scale speaker verification in the wild showed beats i-vector plus PLDA on unconstrained speech (80.5% vs. 60.8% top-1 identification). We keep that deep front end and change only the backend. That paper also found that removing per-frequency variance normalization dropped accuracy to 63.5%. The streaming segmentation work Multi-latency look-ahead for streaming speaker segmentation had to strip out instance normalization because it reads future frames, and it substituted gain augmentation. Together these two results motivate the question of what a causal normalization should be. Our answer is the filtered channel posterior. Subtracting $G\,\mathbb{E}[y_t \mid e_{1:t}]$ is a running normalization that, unlike a moving average, does not absorb a long-talking speaker into the "background."
The second component borrows the principle of Development of the primary CRIM system for the NIST 2008 speaker recognition evaluation. That system integrated over the posterior of the channel factor rather than plugging in a point estimate, which mattered most on 10-second trials. Our per-window decisions marginalize over $y_t$ in the same way, and the filter supplies that posterior at negligible cost because $y_t$ is low-dimensional. Following that paper's finding that jointly trained loadings contaminate the speaker subspace, we learn $F$ and $G$ on disjoint data. We train $G$ on segments with known simulated channel changes, using noise, reverberation and codec augmentation in the style used to train JPLDA.
The third component is online adaptation of what genuinely drifts: the drift covariance $Q$, the mean $\mu$, and the proposal over speaker assignments. Recursive Learning of Asymptotic Variational Objectives gives a recursive, particle-based way to learn both model parameters and a proposal from a single pass over a stream, at linear particle cost. So far it has been tested only on low-dimensional synthetic state-space models. We use it for the switching part of our model, and we compare it against plain recursive maximum likelihood with a bootstrap proposal. If the cheaper method matches it, we will report that result and adopt the cheaper method. The loading matrices stay frozen, because their adaptation from unlabeled data is exactly where pseudo-labels can reinforce errors.
The fourth component is the evaluation protocol. Every system in the comparison receives audio strictly in order. No statistic may use frames beyond an allowed look-ahead $\lambda \in \{0, 0.5, 1\}$ seconds, a range taken from the streaming segmentation work. Global normalization and second passes are disallowed. Labels never feed back into the running system. This is the speech analogue of the "time travel" critique in Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking, which showed in another domain that models scored on data they could effectively see in hindsight collapse when evaluated forward. We also release the causal-audit harness, which flags any component that touches future frames.
Baselines
The naive reference is standard PLDA scoring on causal sliding-window embeddings, with no channel compensation beyond the PLDA within-speaker covariance, followed by the same online clustering our system uses. It gets the same embedding extractor, the same window length, and the same look-ahead. If the stream contains no real channel drift, the full method should collapse to this baseline. The gap between the two therefore measures how much drift the benchmark actually contains.
The strongest causal competitor is causal running normalization of the embeddings: an exponential moving average of mean and variance, with its decay rate tuned on the same development data and the same search budget as our $Q$. This is the obvious engineering answer to the normalization problem. It costs essentially nothing, and it is the closest cheap approximation to our idea.
The strongest existing segmentation system is the multi-latency causal model of Multi-latency look-ahead for streaming speaker segmentation, paired with incremental clustering, run at the same three look-aheads on AMI. That paper reports roughly 100μs computational latency and states that 500ms look-ahead "closes most of the gap" in diarization error to its offline model. However, no reported numeric DER was found on Lacuna for it, so we will rerun it under our protocol.
For an unsupervised adaptation competitor we run VB-MAP from Unsupervised Bayesian Adaptation of PLDA for Speaker Verification, which reports 10–16% relative EER improvement on NIST SRE18 CMN2. Here it is re-run every $k$ seconds on a buffer of past audio only, with $k$ chosen so its compute per hour of audio matches ours.
As non-causal upper references we report the offline two-pass factor-analysis segmentation of Factor analysis for speaker segmentation and improved speaker diarization, which reports a speaker error rate on COST278 falling from 10.7% to 9.8% with its second pass, and offline JPLDA with oracle condition labels. Beating our causal baselines and closing at least half the gap to these offline references at 500ms look-ahead would count as success.
Three ablations isolate the key ingredient. In the first, the channel state is frozen ($Q = 0$), which reduces the model to static channel compensation. In the second, the speaker and channel terms are merged into one drifting offset, which reduces it to a learned moving average. In the third, we supply the true channel-change times, which bounds how much the filter loses by having to infer them.
Why not just…?
Why not just use an exponential moving average to normalize embeddings causally? This is the real threat, and we treat it as one. A moving average cannot tell a long single-speaker stretch from a channel shift. It will therefore partly subtract the speaker's own identity during monologues and mis-normalize a speaker who returns after a channel change. Our model predicts gains concentrated in exactly those two situations. We will stratify errors by turn length and by "speaker recurs across a channel change." If the moving average matches us in both strata, the structured model has not earned its complexity, and we will say so.
Why not just buffer 10–30 seconds and run offline factor analysis on each chunk? Chunking adds latency equal to the chunk length, far beyond the sub-second budgets considered here. Short chunks are also where factor estimates are weakest: the CRIM system reports 13.4% EER on 10-second trials against 1.4% on full conversations. Our periodic VB-MAP baseline is precisely this alternative at matched compute, so the experiments settle it directly.
Why not just train a causal end-to-end network with heavy channel augmentation? Augmentation teaches invariance to channels seen in training but gives no mechanism to track a new channel as it appears. The causal segmentation work itself reports an accuracy loss after dropping normalization. We include that model as a baseline. We also test on held-out codecs and rooms absent from augmentation, where a tracking model should hold up better than a purely invariant one.
Proposed Experiments
We evaluate on three benchmarks. On AMI meetings we measure diarization error at $\lambda \in \{0, 0.5, 1\}$ s. On COST278 broadcast news we measure boundary recall and speaker error rate, so results are comparable to the offline factor-analysis segmentation figures. The third benchmark is a constructed drift stream. We build it by concatenating VoxCeleb utterances into multi-speaker sessions and applying channel changes at known times (codec switches, room changes, SNR ramps), some with held-out codecs. On this stream we report per-window verification EER against enrolled speakers, along with time-to-recover after a channel change. Every system reports real-time factor and per-window latency. Embedding extractor, windows, and tuning budget are shared across all systems.
The decisive check comes from the constructed stream, where channel changes are known. If the filtered channel state does not move at the true change points, and the advantage over the moving average does not appear in the recurrence and monologue strata, then the speaker/channel separation is not happening, and the claim fails regardless of aggregate numbers. A second check is AMI with essentially stationary channels. There our method should neither win nor lose much against plain PLDA, which confirms that gains elsewhere come from drift tracking rather than from extra tuning.
References
- A Generalization of PLDA for Joint Modeling of Speaker Identity and Multiple Nuisance Conditions
- Voxceleb: Large-scale speaker verification in the wild
- Multi-latency look-ahead for streaming speaker segmentation
- Development of the primary CRIM system for the NIST 2008 speaker recognition evaluation
- Recursive Learning of Asymptotic Variational Objectives
- Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking
- Unsupervised Bayesian Adaptation of PLDA for Speaker Verification
- Factor analysis for speaker segmentation and improved speaker diarization