Evidence-Weighted Target Speaker Extraction for Short Enrollments

Voice-based identity checks increasingly happen in messy audio: a phone call with someone talking in the background, or a smart speaker in a busy kitchen. The standard fix is target speaker extraction. A neural network is given a sample of the enrolled person's voice and pulls that voice out of a mixture before the verifier makes its decision. Every such system in the literature assumes the enrollment sample is long and clean. In practice it is often one or two seconds. At that length the reference contains only a handful of speech sounds, so the resulting voice summary mostly encodes what was said, not who said it. An extractor conditioned on that summary will confidently remove parts of the target's voice it has simply never heard. It will also invent the target's voice when the target is absent.

We propose to borrow the oldest trick in speaker recognition for this new setting. The classical GMM-UBM approach is a Gaussian mixture model that is first fit to a large population of speakers (the "universal background model") and then updated toward each enrolled speaker. That update is a Bayesian average. Each region of acoustic space moves toward the speaker only in proportion to how many frames of that speaker it actually observed, and otherwise stays at the population default. We would represent a short enrollment as such a region-by-region adapted profile, together with its per-region evidence counts. We would then make both the extractor and the verifier use those counts. Where enrollment evidence is thin, the extractor suppresses less, and the verifier scores only the regions that enrollment and test have in common. The idea, in two sentences: tell the extraction network how much it actually knows about the target speaker in each part of acoustic space, and have it back off to gentle filtering where it knows little. This should shrink both the distortion and the hallucination errors that short enrollments cause under overlapping speech.

Core Claim

Short-enrollment failure under overlap is a problem of misplaced confidence, not only one of missing information. A single enrollment embedding gives the extractor no way to tell "this speaker never produced this sound in enrollment" apart from "this sound does not belong to this speaker." The extractor therefore treats unobserved phonetic regions as interference and zeros them out. This is the same risk-averse over-suppression that Neural Kalman Filtering for Speech Enhancement documents for regression-trained masks, but here it is driven by enrollment gaps rather than noise.

We claim that conditioning on per-component, count-weighted statistics removes this confound. Coupling mask aggressiveness to enrollment evidence, and restricting scoring to regions covered by both enrollment and test, should give verification gains that grow as enrollment shrinks. Those gains should concentrate on enrollment/test pairs with little phonetic overlap and should vanish when enrollment is long. That dependence pattern is what separates the claim from "a better extractor."

Relation to Prior Work

Three systems motivate the gap by solving overlap-robust verification while assuming ample enrollment.

None of the three represents how much the enrollment covers. We keep their overall structure but replace the single conditioning vector with an evidence-weighted one.

Two works supply the method.

  • Phone-centric local variability vector for text-constrained speaker verification shows that on short text-constrained speech, scoring only the phones present in both enrollment and test beats global scoring. It also shows that concatenating unobserved phone vectors hurts. That paper relies on a supervised ASR phone aligner and clean single-speaker audio. We replace the phones with unsupervised mixture components, which makes the method text-independent, and we apply the matching under overlap.
  • Integrating Uncertainty into Neural Network-based Speech Enhancement shows that softening a filter where posterior variance is high reduces distortion. We reuse that logic, but our uncertainty comes from enrollment coverage rather than from noise.

Two findings constrain the design.

  • EA-ASP reports that feeding a rich enrollment representation into the verifier causes "identity leakage": the network copies enrollment identity to the output and ignores the test speech. EA-ASP had to bottleneck the enrollment to two dimensions. A full adapted profile invites exactly this leakage, so our design must guard against it.
  • Auxiliary Loss Function for Target Speech Extraction and Recognition with Weak Supervision Based on Speaker Characteristics shows that requiring the extracted target and the residual to sum back to the mixture curbs over-filtering, at least for ASR. We adopt that constraint and test it for verification, which that paper never did.

Research Program

Background model. We first fit a background mixture model with K components (K of 64 and 256 will be compared) on frame features from a frozen intermediate layer of WavLM. That model was pre-trained with simulated interfering speakers, so its frames are somewhat overlap-aware. We will also run an MFCC-based background model as a control.

Enrollment profile. For an enrollment clip, we compute each component's soft frame count $n_m$ and mean feature $E_m$. We then form the classical adapted offset with a fixed relevance factor $r$:

$$\alpha_m = \frac{n_m}{n_m + r}, \qquad d_m = \alpha_m\,(E_m - \mu_m^{UBM}).$$

Here $\mu_m^{UBM}$ is the background mean of component $m$. When a component received no enrollment frames, its offset is exactly zero and $\alpha_m=0$ marks it as unknown.

Extractor conditioning. The extractor is a Conv-TasNet-style time-domain network, matching TASE-SVNet and the A-MAP work. It receives the stacked pairs $(d_m, \alpha_m)$. To avoid identity leakage, these pass through a narrow projection shared across components, rather than being concatenated whole.

Evidence-dependent masking. For each mixture frame $t$, we compute an evidence score $c_t=\sum_m \gamma_{tm}\alpha_m$, where $\gamma_{tm}$ is the frame's posterior probability of belonging to component $m$ under the background model. The predicted mask is then interpolated toward a mild, speaker-independent suppression mask as $c_t$ falls. This interpolation is the explicit link to the intelligibility–quality trade-off: aggressiveness is spent only where the extractor has evidence to justify it.

Training. Training uses a scale-invariant signal-to-distortion loss on the extracted target, the mixture-consistency term from Zmolíková et al., and TASE-SVNet-style target-absent mixtures that must produce near-silence. Enrollment clips are randomly cropped to 0.5–4 s.

Verification. The verifier is a standard embedding network run on the extracted speech. We will compare two scoring modes. The first is a plain cosine score. The second is coverage-matched scoring, in which the test clip's own per-component statistics are computed and the enrollment–test similarity is weighted by $\min(\alpha_m^{enr},\alpha_m^{test})$. This is an unsupervised analogue of content-matched scoring.

Baselines

The naive reference points are the same verifier applied directly to the mixture with no extraction, and the same verifier applied to the clean target speech (a clean-speech upper bound). For comparison, SE-SV reports 21.80% EER without extraction and 3.0% EER on clean speech on its WSJ0 mixtures. Those numbers come from full-length enrollment. We will rerun both conditions under our short-enrollment protocol rather than borrow them.

The strongest existing methods are TASE-SVNet and EA-ASP.

  • TASE-SVNet reports 5.75% EER on its heavily overlapped Multi-Spk task, down from a 15.91% TDNN baseline.
  • EA-ASP-M reports 5.21% EER on simulated two-speaker VoxCeleb1 overlap, versus an 11.16% baseline. It also degrades from 1.13% to 1.60% on clean single-speaker trials.
  • VoiceExtender (VoiceExtender) reports 9.32% EER at 0.5 s versus a 17.29% baseline on clean single-speaker VoxCeleb1. It is the strongest short-duration method but was never tested with overlap.
  • No reported result was found on Lacuna for any system that combines overlap with short (<2 s) enrollment. The short-enrollment operating points for all of these baselines therefore have to be produced by us.

We will reimplement EA-ASP-M and a TASE-SVNet-style extractor with nontarget training. Both will use the same WavLM features, the same verifier backbone, the same training mixtures, and the same random enrollment-cropping augmentation. Each gets a matched hyperparameter search budget. A reimplementation will count as faithful if, with full-length enrollment, it lands within roughly one EER point of the published figure.

Two ablations isolate the key ingredient.

  • Scalar shrinkage. The first keeps a single enrollment embedding and shrinks it toward the population mean with one global count-based coefficient. This removes the per-component structure but keeps the Bayesian averaging.
  • No evidence gating. The second feeds the per-component profile but removes both the evidence-dependent mask interpolation and the coverage-matched scoring. This tests whether the benefit comes from representation alone.

Beating the baselines means a statistically significant EER and minDCF reduction over the augmented TASE-SVNet-style and EA-ASP-M systems at 0.5 s and 1 s enrollment. The target-absent false-alarm rate must be no worse, and the gap must shrink toward zero at 4 s or more of enrollment.

Why not just…?

Why not just train an existing extractor with randomly cropped short enrollments? This is the real threat. Duration augmentation alone might teach a single embedding to be cautious. For that reason, every baseline above is trained with exactly the same cropping, so our method gains nothing from augmentation that the baselines lack. If the augmented single-embedding system matches ours on phonetically disjoint pairs, the claim that the confound is structural is wrong. We would then report that result instead of the method.

Why not just skip extraction and use enrollment-aware pooling? EA-ASP is lighter and strong under full enrollment. However, its two-dimensional enrollment bottleneck discards exactly the coverage information we argue matters, and it already pays a clean-speech penalty (1.13% to 1.60% EER). We will measure whether that penalty and its overlap gains worsen as enrollment shortens. We will also test whether adding coverage-matched scoring to EA-ASP alone recovers most of our gain. If it does, the cheaper hybrid is the better recommendation.

Why not just lengthen the enrollment with a generative model first? VoiceExtender roughly halves short-utterance EER on clean speech. It needs iterative diffusion sampling, however, and the synthetic frames can carry voice cues the real speaker never produced, which is dangerous in an extractor that then hunts for those cues in a mixture. We will run VoiceExtender-extended enrollment feeding the baseline extractor, and report EER, target-absent false alarms, and wall-clock latency. We will not argue this alternative away. If it wins on accuracy, the comparison becomes a latency–accuracy question.

Proposed Experiments

Primary benchmark. The primary benchmark follows the EA-ASP protocol: simulated two-speaker VoxCeleb1 mixtures with overlap ratio up to 0.5 and SNR from −3 to 3 dB, training on VoxCeleb2. Enrollment is truncated to 0.5, 1, 2, and 4 s, and test clips to 2 s and full length.

Secondary benchmarks. We will use SE-SV-style WSJ0 two-talker mixtures at 0–5 dB signal-to-interference ratio, and a 1–3 speaker overlap condition with a target-absent subset in the spirit of TASE-SVNet. RSR2015 Part III serves as a text-constrained check against the content-matched scoring of Chen et al., whose sph-lv reports 4.192% male and 5.407% female EER.

Metrics. We report EER, minDCF, and the false-alarm rate on target-absent mixtures. We also report SI-SDR of the extracted speech, to test whether signal-fidelity gains track verification gains at all, since the research file flags that they may decouple. Finally, we report clean single-speaker EER, to measure the filtering penalty.

Decisive checks. Three checks decide the question.

  1. Using background-model coverage, we split trials by enrollment–test component overlap. The proposal predicts its advantage over the augmented single-embedding baseline is largest on low-overlap pairs and near zero on high-overlap pairs.
  2. The advantage should decline monotonically with enrollment length.
  3. On target-absent mixtures, evidence gating should lower false alarms relative to nontarget training alone, because a thin enrollment can no longer license confident extraction.

If gains appear uniformly across overlap bins and durations, the improvement is generic capacity rather than the mechanism proposed here, and we would report it as such.