Shifting the Enrollment, Not the Mixture: Speaker Verification When the Target Shouts Over Someone Else

Speaker verification systems answer a simple question: is the person in this test recording the same person who enrolled earlier? They fail in two well-documented ways. The first is when the speaker changes how they physically produce their voice, for example shouting where they enrolled speaking normally. The second is when another person talks at the same time. These failures tend to occur together. People shout in emergencies, in crowds, and in arguments, which are exactly the situations where other voices are present. Yet every method we could find handles only one of them.

This note proposes a fix aimed at the place where the two failures interact. The best overlap-robust verifiers use the enrolled voice as a "key" that tells the network which frames of the mixture belong to the target. The best shout-robust methods estimate how shouting displaces a voice representation and subtract that displacement from the test recording. When the test recording is a mixture, that subtraction stops making sense. Our idea is to apply the shouting displacement to the clean enrollment instead. We build several "what this person would sound like shouting" versions of the enrollment key, use each one to pick out the target from the mixture, and average the resulting scores over how likely each effort level is. This treats vocal effort the way cosmologists treat intrinsic galaxy alignment: as a nuisance that imitates the signal and should be marginalized over, not ignored and not point-estimated.

Core Claim

The claim is that vocal-effort compensation must move from the test side to the enrollment side once overlap is present. Doing so recovers most of the gap between "shouted, overlapped" and "neutral, overlapped" verification error, with no need for parallel shouted training mixtures.

The motivation comes from two results that conflict when combined. Enroll-Aware Attentive Statistics Pooling for Target Speaker Verification passes the enrollment embedding through a two-dimensional bottleneck. The bottleneck output produces frame-level attention or soft masks that suppress the interfering talker during pooling. This cuts multi-speaker equal error rate (EER, the operating point where false accepts equal false rejects) from 11.16% to 5.21%, but it assumes the enrollment and the test share neutral effort. Separately, Analysis and classification of speech mode: whispered through shouted and Shouted and whispered speech compensation for speaker verification systems show that speaker embeddings cluster by vocal mode rather than identity, and that effort mismatch inflates error by over 114%. Our hypothesis is that a neutral key will lock onto the wrong frames when the target shouts. We expect it to favor the interferer, whose voice may sit closer to the target's neutral profile than the target's own shouting does.

The method comes from Improved Vocal Effort Transfer Vector Estimation For Vocal Effort-Robust Speaker Verification. That paper models a shouted embedding as the neutral one plus an additive "transfer vector" in a 16-dimensional principal-component space of WavLM/ECAPA-TDNN embeddings, and estimates the vector from the test observation. Its estimator needs a single-speaker test embedding, and a mixture does not provide one. The enrollment, however, is clean and single-speaker, and an additive shift can be applied there in either direction. Our contribution is this inversion, together with a marginalization over effort hypotheses.

Research Program

The system starts from the backbone used for transfer vectors: a WavLM front-end feeding an ECAPA-TDNN embedding extractor. We chose WavLM partly because WavLM was pre-trained with simulated interfering speakers, so it already provides some resistance to overlap. We replace the pooling layer with enrollment-aware masked pooling, as in the EA-ASP paper.

From speaker-disjoint training folds of the Finnish Shouted-Normal Corpus, we fit a small Gaussian mixture to per-speaker shouted-minus-normal differences in the 16-dimensional principal-component space. The mixture means $v_1,\dots,v_K$ become effort hypotheses. We also include $v_0 = 0$ for "no change." For a whispered extension we fit the same kind of mixture on CHAINS.

At test time, enrollment embedding $e$ yields $K+1$ shifted keys $e_k = e + P^\top v_k$, where $P$ is the principal-component projection. Each key drives one masked pooling pass over the mixture, producing test embedding $t_k$ and cosine score $s_k = \cos(t_k, e_k)$. The final score is

$$ S = \log \sum_{k=0}^{K} \pi_k(x)\, \exp(\beta s_k), $$

where $\pi_k(x)$ is the probability of effort hypothesis $k$ given the mixture $x$, and $\beta$ is a temperature tuned on development data. We obtain $\pi_k(x)$ from a lightweight effort classifier trained on mixtures, or set it to the training prior as an ablation. Shouting detection on clean x-vectors is easy, at 98.11% accuracy per Shouted Speech Compensation for Speaker Verification Robust to Vocal Effort Conditions. Whether detection survives overlap is itself something we will measure.

The analogy to cosmology is a design principle, not an import of technique. Cosmological constraints with deep learning from KiDS-450 weak lensing maps obtains its constraints by marginalizing over the alignment amplitude $A_{\mathrm{IA}}$ rather than fixing it. Dark Energy Survey Year 3 results: Cosmology from cosmic shear and robustness to modeling uncertainty shows that the choice of nuisance model can move headline conclusions. The parallel lesson is that point-estimating effort, the way a single transfer vector does, is brittle when the estimate comes from a corrupted observation.

Depthwise separable convolution plays a narrow engineering role. Xception factorizes a convolution into a per-channel filter followed by a 1×1 channel-mixing step, which cuts parameters roughly ninefold for 3×3 kernels. Running $K+1$ masked pooling passes multiplies the cost of the mask generator, so we implement that generator as depthwise-separable 1-D convolutions over time. This keeps the added parameters small, which matters when the effort data covers only 22 speakers. We will check that this choice does not cost accuracy relative to a dense mask head. We do not claim it is a contribution beyond feasibility.

Baselines

The naive reference point is the WavLM/ECAPA-TDNN system with ordinary attentive pooling. Its scores come straight from the mixture embedding against the neutral enrollment, with no overlap handling and no effort handling.

The strongest overlap competitor is EA-ASP with soft masks, reimplemented on the same backbone and trained on the same VoxCeleb-based mixtures. On neutral simulated mixtures the EA-ASP paper reports 5.21% multi-speaker EER, against 11.16% for its baseline. We will first reproduce this regime as a sanity check.

The strongest effort competitor is the MMSE transfer-vector estimator applied to the mixture embedding. It reports normal-versus-shouted EER of 17.74%, down from 21.76%, on clean single-speaker Finnish data. We run it with identical principal-component dimension and training folds.

No reported result exists for any system on the joint shouted-plus-overlap condition, so we must construct that test set ourselves. For the joint condition, beating the baseline means a statistically reliable EER reduction over the best of these systems, measured with speaker-level bootstrap intervals. All systems share the same backbone, training mixtures, folds, and hyperparameter search budget.

Three ablations isolate the key ingredient. The first is $K=0$, which reduces to EA-ASP and tests whether the shift matters. The second uses a single mean transfer vector, which tests marginalization against point estimation. The third applies the shift only in scoring and not in the mask key, which tests whether the gain comes from better target selection or merely from score correction.

Why not just…?

Why not just separate the mixture first, then apply transfer-vector compensation to the separated target stream? This is the most natural modular answer, and we treat it as a serious threat. The EA-ASP paper argues that a separate separation network adds substantial overhead. End-to-End Neural Speaker Diarization with Permutation-Free Objectives shows modular pipelines losing to joint models on simulated overlap: 28.77% versus 12.28% diarization error rate. Neither result concerns shouted speech, though. A separator trained on neutral speech may also distort shouted voices, whose spectral tilt and pitch fall outside its training range. We will run a separation-then-MMSE pipeline and report its accuracy and compute cost against ours. If it matches us, the honest conclusion is that enrollment-side shifting buys efficiency rather than accuracy.

Why not just train EA-ASP on mixtures that include shouted targets? This multi-condition training is the cheapest route, and it might simply work. The risk is data. The shouted corpus has 22 speakers, and Improved Vocal Effort Transfer Vector Estimation needed a 16-dimensional projection just to keep its estimator stable. A network learning the joint condition from such data may overfit to those speakers. We include this baseline at matched data and test it on held-out speakers. We also run a combined variant, multi-condition training plus enrollment shifting. If multi-condition training alone closes the gap, we will report that and drop the claim.

Why not just detect shouting in the mixture and apply one transfer vector? This is our single-vector ablation with a detector gate. It fails if detection degrades under overlap, or if one mean shift poorly represents speaker-specific shouting. Both are measurable: we report detector accuracy as a function of signal-to-interference ratio, and we compare single-vector against marginalized scoring.

Proposed Experiments

The test set mixes Finnish Shouted-Normal and CHAINS target utterances with VoxCeleb1 interferers. It follows the EA-ASP protocol of signal-to-interference ratios from −3 to 3 dB and overlap up to 0.5. Enrollment is always neutral. Test effort is neutral, shouted, or whispered, and in one condition the interferer also shouts. Because the corpora are small, all effort statistics are estimated in leave-speakers-out folds, and EER is reported with confidence intervals over speakers.

The first decisive check tests the premise directly. Because we know the clean sources of each mixture, we can measure how much of EA-ASP's attention mass falls on frames where the target dominates. We compare neutral and shouted targets. If attention on the target does not drop markedly for shouted targets, then the key-mismatch hypothesis is wrong, and any gain we see must be explained as score correction instead.

The second check compares the joint-condition EER of our method against the three baselines and the ablations. We also report clean single-speaker EER, because EA-ASP already raises it from 1.13% to 1.60%. The included zero-shift hypothesis should prevent further loss there.

The third check reports detector accuracy and the marginalization weights as overlap increases. This shows whether the method degrades gracefully when the effort cue itself is buried in the interfering voice.