Who Else Is in the Enrollment? Structured-Impostor Reliability for Target Speaker Extraction

Target speaker extraction pulls one person's voice out of a recording of several people talking at once. The system is told whom to extract through a short "enrollment" clip of that person speaking. Nearly every extraction system treats that clip as a clean, trustworthy example of the target voice. That is the speech version of what imitation learning calls the expert optimality assumption: the belief that every demonstration is worth copying. In real meetings and calls the assumption fails. Enrollment clips are cut from the same recording and carry crosstalk. The crosstalk usually comes from the very people who also interfere in the mixture, and those people often sound like the target. A corrupted clip then yields a speaker fingerprint that sits between the target and a similar-sounding interferer, which is exactly the failure behind "target confusion."

We propose to treat each enrollment frame as a demonstration of uncertain quality and to estimate, frame by frame, how much it should be trusted. The key departure from imitation-learning methods that score demonstrator expertise is the model of what a bad frame is. Those methods model a bad demonstration as random noise around the correct behavior. We instead model bad enrollment frames as coming from a specific, coherent impostor voice. The impostor's identity is estimated jointly from the enrollment and from the mixture being separated. The claim to test is this: when corruption comes from an acoustically similar speaker, an explicit structured-impostor reliability model recovers most of the gap to clean enrollment. It should also clearly beat both random-noise reliability models and plain learned attention over enrollment frames, and the advantage should grow as the two voices become more similar.

Background and the Gap

Two strands meet here. On the imitation side, Imitation Learning by Estimating Expertise of Demonstrators (ILEED) learns a state-dependent expertise score that blends an optimal policy with unstructured noise. It outperformed behavioral cloning, GAIL, and IRIS in 21 of 23 settings. Its own open problem is that it cannot separate systematic errors from intent, because it only models suboptimality as random variance. Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality (CAIL) learns per-transition confidence weights through a two-level optimization. It needs a small ranked validation set, and speech has no natural equivalent. DualCOIL pulls a policy toward good data and pushes it from bad data. It shows that capping the push weight at $\alpha \le 1$ keeps the objective convex and prevents drift into unseen failures. These papers supply the method. ILEED's noise assumption is the specific limitation we target, and DualCOIL's bounded repulsion is a design constraint we adopt.

On the speech side, Target Confusion in End-to-end Speaker Extraction: Analysis and Approaches shows that confusion is driven mainly by embedding bias, meaning similar voices map to nearby fingerprints. It does not study enrollments corrupted by a similar speaker, which plausibly makes that bias worse. Multi-Level Speaker Representation for Target Speaker Extraction attends over enrollment frames and spectral bases, and it reports large gains with clean enrollments. Its frame-level cross-attention is the closest existing mechanism to soft frame weighting, but it has no notion of a competing identity inside the enrollment. Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments handles noisy enrollments by pairing a target-active clip with a target-silent clip and canceling shared content. It therefore requires a negative clip and boundary labels, and it assigns no per-frame reliability. Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling trains on synthetically corrupted embeddings. It gains robustness by data augmentation alone, with no explicit model of which cues are bad. These four papers motivate the problem and supply the backbones and baselines. None of them models the enrollment corruption as a structured second identity that is shared with the mixture.

Core Claim

Let $e_t$ be the speaker-encoder embedding of enrollment frame $t$. We model each frame as drawn either from the target voice or from an impostor voice:

$$p(e_t) = \rho_t\, \mathcal{N}(e_t; \mu_T, \Sigma) + (1-\rho_t)\, \mathcal{N}(e_t; \mu_I, \Sigma).$$

Here $\rho_t \in (0,1)$ is the frame's reliability, $\mu_T$ is the target prototype, and $\mu_I$ is the impostor prototype. Both prototypes share a covariance $\Sigma$.

ILEED's second component is uniform noise. Ours is a coherent voice, and $\mu_I$ is tied to evidence from the mixture. An amortized network predicts $\rho_t$ from $e_t$, from local overlap features, and from cross-attention against mixture frames. Frames that resemble a voice present in the mixture but absent from the rest of the enrollment are pushed toward the impostor component.

The extractor is conditioned on the reliability-weighted prototype $\hat{\mu}_T = \sum_t \rho_t e_t / \sum_t \rho_t$. It is also conditioned on $\mu_I$ as an explicit "not this person" signal. A bounded repulsion term in the training loss pushes the output embedding away from $\mu_I$ with weight $\alpha \le 1$. In deep networks this cap is a stability heuristic borrowed from DualCOIL's anchoring argument, not a convexity guarantee.

The prediction follows from the setup. A random-noise model can down-weight frames that look unusual. It cannot down-weight frames that look like a perfectly plausible but wrong speaker, because those frames are not unusual. The structured impostor component is built for that case.

Research Program

We will build corrupted-enrollment versions of standard benchmarks. Into each clean enrollment we mix an interfering speaker at controlled overlap ratios (0–80%) and controlled signal-to-interference ratios. The interferer comes in two conditions. In the "shared" condition it is the same person who interferes in the mixture, which mimics meeting crosstalk. In the "unrelated" condition it is a different speaker. We bin every test pair by the cosine similarity between the clean target and interferer embeddings, so that similarity is a measured axis rather than a gender proxy.

All extraction runs use one fixed backbone: the multi-level Band-Split RNN extractor of Multi-Level Speaker Representation for Target Speaker Extraction. The only thing that changes between systems is how the enrollment is turned into conditioning. The reliability network and the extractor are trained jointly with the scale-invariant signal-to-distortion ratio (SI-SDR) extraction loss, the bounded repulsion term, and the mixture likelihood above as an auxiliary loss. Corruption-rate and similarity schedules are randomized during training.

We will also analyze whether the learned $\rho_t$ recovers the known frame-level corruption labels, which exist because the corruption is synthetic. We will check whether $\mu_I$ lands near the true interferer's clean embedding.

Baselines

The naive reference point is the same backbone given corrupted enrollments and trained only on clean ones. This is the unmodified expert-optimality assumption. The ceiling is the same backbone with oracle clean enrollments. On clean Libri2Mix the cited paper reports SI-SDRi of 15.91 dB and extraction accuracy of 97.0% for this configuration (Multi-Level Speaker Representation for Target Speaker Extraction). We will reproduce rather than borrow that figure. For the corrupted-enrollment setting itself, we found no reported result on Lacuna, so every number there will come from our own runs.

The strongest competitor is the same backbone, with its frame-level cross-attention, retrained on exactly our corrupted-enrollment training distribution. This is the augmentation recipe of Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling applied to this extractor. It is the strongest competitor because learned attention, given the same data, could in principle learn implicit reliability weighting. The proposal has to beat it on equal data, equal parameter count (within 5%), equal training steps, and equal hyperparameter search budget.

The second strong competitor is the positive/negative enrollment system of Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments. That paper reports a monaural SI-SNRi gain of over 2.1 dB against a prior method called TCE, but no absolute value is reported in our file. We will give this system a target-silent negative clip, which our method does not receive, so the comparison is deliberately favorable to it.

We also include the post-filter with linear subtraction from Target Confusion in End-to-end Speaker Extraction: Analysis and Approaches. On clean Libri2Mix it reports 13.88 dB SI-SDRi when combined with prototypical metric learning. We test it in two-speaker conditions only, since the subtraction is limited to that case.

Two ablations isolate the key ingredient. The first replaces the impostor Gaussian with an ILEED-style broad noise component and keeps everything else fixed. The second removes the mixture cross-attention, so that $\mu_I$ is estimated from the enrollment alone.

Beating a baseline means a statistically reliable SI-SDRi gain, tested with paired bootstrap over test utterances, together with a lower target-confusion rate. The gain must hold in the top similarity bins and at corruption rates of 40% and above. That is where the hypothesis says the methods should diverge.

Why not just…?

Why not just detect overlap and throw away overlapped enrollment frames? Hard filtering is cheap and may work when corruption is sparse. It fails in two ways we can measure. First, at high overlap ratios it discards most of the clip. Second, an overlap detector will itself struggle when the two voices are similar, since that is exactly what makes the clip hard. We include an oracle-overlap version of this filter. If even the oracle filter matches our method, the structured model adds nothing beyond knowing where corruption is, and we will report that result.

Why not just train the existing attention-based extractor on corrupted enrollments? This is the real threat, and it is the strongest baseline above. Implicit attention trained on randomized corruption may learn to down-weight atypical frames. Our hypothesis is that it will not reliably down-weight frames of a plausible wrong speaker. We also expect it to generalize worse to corruption rates and similarity levels outside its training range. The test is to hold out the highest-similarity bins and the 60–80% corruption rates from training. If attention pooling matches us there too, the claim is falsified.

Why not just use the positive/negative enrollment comparison? It needs a separately segmented target-silent clip and timing annotations, although the cited paper shows it tolerates boundary errors of about 0.3 s. It cancels shared content rather than identifying a competing identity. We evaluate it with its negative clip supplied. If it wins despite that extra input, we will report the gap as the price of not having a negative clip.

Why not just use a bigger speaker encoder? Target Confusion in End-to-end Speaker Extraction identifies embedding bias toward similar speakers as the main cause of confusion. A stronger encoder still averages a contaminated clip. We will swap encoders under every method to confirm that the ranking of methods is stable.

Proposed Experiments

The benchmarks are Libri2Mix, clean and noisy, from LibriMix, and WSJ0-2mix-extr, using its same-gender subset, where SpEx+ reports its largest gains. Each comes in shared- and unrelated-interferer corrupted-enrollment variants. We add a three-speaker Libri3Mix variant to test the regime where subtraction cannot apply. Finally, LibriCSS tests whether the method transfers to long recordings when enrollments are taken from the recording itself. There we replace the K-means profiles of Continuous Speech Separation Using Speaker Inventory for Long Recording with reliability-weighted prototypes and measure diarization error rate and concatenated minimum-permutation word error rate. Metrics are SI-SDRi, extraction accuracy, and target-confusion rate, defined as the fraction of outputs whose embedding is closer to an interferer than to the target. All three are reported per similarity bin and per corruption rate.

Three checks decide the outcome. The first: the gain over the corruption-trained attention baseline should increase monotonically with target–interferer similarity. A flat gain would mean the method is just a better general robustness trick, not a fix for structured corruption. The second: the noise-component ablation should lose most of the gain in high-similarity bins while staying competitive in low-similarity bins. This would show that the structured impostor model, not the reliability weighting itself, does the work. The third: the advantage should be larger in the shared-interferer condition than in the unrelated one, and the no-mixture-attention ablation should close that difference. This would confirm that identifying the impostor from the mixture is the mechanism. If all three hold and the method recovers a large fraction of the oracle-clean gap, the case is made that the expert-optimality assumption on enrollments is a distinct, fixable source of target confusion.