Target speaker extraction pulls one person's voice out of a crowd. Instead of separating every talker at once, the model is given a short "enrollment" recording of the person you want and returns only that voice. This design is what lets extraction scale to any number of background talkers. It also creates a single point of control. If an attacker adds a small, inaudible perturbation to the enrollment clip or to the mixture, the model may not just degrade. It may switch to extracting someone else. Speaker embedding networks are already known to be easy to fool, yet nobody has measured what this does to an extraction system, how the damage grows as the room gets more crowded, or which internal part of the network the attack actually exploits.
We propose to answer the "where" question first, then use the answer to build a cheaper defense. The first step is a diagnosis. We attack each pathway of an extraction network separately, then use activation patching, which copies clean internal activations into an attacked run, to see which component accounts for the failure. The second step is the defense. We apply adversarial training only to that component, scheduled from weak to strong attacks. We test both steps with one to three interfering talkers, because a redirection attack has more voices to redirect toward as the crowd grows.
Core Claim
Our hypothesis is that adversarial fragility in target speaker extraction is concentrated in the pathway that fuses the enrollment clue with the mixture, not in the mask estimator that shapes the output. The resulting failure is a "redirection": the output locks onto an interferer. We expect redirection to become more likely as the number of interferers grows and as interferers sound more like the target.
If this holds, adversarial training restricted to the clue-and-fusion pathway, with an easy-to-hard schedule over attack strength, should match the robustness of adversarially training the whole network. It should also lose much less clean extraction quality. Two outcomes would refute the claim: patching the fusion activations recovers only a minority of the attack-induced loss, or targeted training is no better than whole-network training on the clean-versus-robust trade-off.
What Is Known and What Is Not
Three papers motivate the problem. x-Vectors Meet Adversarial Attacks benchmarks speaker-verification embeddings under gradient attacks. Iterative attacks push the actual detection cost above 1.0, which is worse than an uninformative system, at 30 dB signal-to-noise ratio for evasion and 50 dB for impersonation, and attacks transfer between architectures. It stops at verification and proposes no defense. Our work asks what the same perturbations do once the embedding conditions an extractor.
Neural Speaker Extraction with Speaker-Speech Cross-Attention Network shows that deep convolutional extractors such as SpEx+ stop using the speaker clue in later blocks. It fixes this with cross-attention, reaching 18.8 dB SDRi on WSJ0-2mix against 17.2 dB for SpEx+. That diagnosis was made only on clean two-speaker data. We test whether attacks re-create the same decoupling.
Target Speaker Extraction with Curriculum Learning orders training examples by target–interferer speaker similarity or by the model's own loss. It improves iSDR on Libri2talker from 12.57 dB to 13.54 dB. It does not test robustness or more than two talkers. Its similarity measure becomes one of our evaluation axes.
Three other papers supply the methods. Curriculum Adversarial Training raises attack strength gradually and mixes earlier attack levels into each batch. On CIFAR-10 it lifts worst-case accuracy from 46.18% to 69.27%, at a 5–6% cost in clean accuracy. We reuse its schedule and batch mixing, but apply them to a regression-style extraction objective and to one component rather than the whole network. Interpretability in the Wild supplies the activation-patching methodology and the faithfulness-style metric; its IOI circuit recovers 94% of model behaviour. Rethinking the Separation Layers in Speech Separation Networks is the template for a parameter-matched component study. It shows that reallocating a fixed depth between multi-output and single-output blocks moves SI-SDR from 9.0 to 9.5 dB.
The curriculum literature contains an apparent contradiction we must face. In vision, curriculum adversarial training costs clean accuracy. In speech, curriculum learning raises clean quality. Our design measures both quantities on the same models, so the contradiction is settled empirically rather than assumed away.
Research Program
We build two extraction models with different fusion styles. The first is a cross-attention extractor in the style of the network above. The second is a concatenation/multiplication-conditioned extractor in the SpeakerBeam style used by Liu et al. Using both tells us whether any localization is a property of extraction in general or of one architecture. Each model is trained on Libri2talker and on LibriMix mixtures with one, two and three interferers, built from Libri2Mix and Libri3Mix sources (LibriMix).
We consider two threat models. In a clue attack, the attacker perturbs the enrollment clip, as when a planted or spoofed enrollment is used. In a mixture attack, the attacker perturbs the mixture, as when a device in the room plays sound. Perturbation budgets are expressed as signal-to-noise ratios from 30 to 60 dB, following Villalba et al. Each attack has two objectives. Untargeted attacks minimize extraction SI-SDR. Redirection attacks maximize SI-SDR with respect to a chosen interferer.
Localization uses activation patching rather than zeroing or mean ablation, because replacing activations with zeros or means can create internal states the network never saw in training. For each component, we copy its clean activations into the attacked forward pass. The components are the speaker encoder, the fusion or cross-attention layers, the early mixture encoder, and the mask estimator. We then report the fraction of the attack-induced SI-SDR loss that is recovered. We also report two-component patches, since the IOI work found backup components that compensate when a primary one is removed, and single-component knockouts miss that.
Hardening applies PGD adversarial training to the parameters of the localized component only, with the remaining weights frozen or trained on clean loss only. The attack strength follows Cai et al.'s schedule from clean to strong, with earlier strengths mixed into every batch. The number of interferers is also raised from one to three across training. That second axis is an ablation, not an assumption.
Baselines
The naive reference point is the same extractor trained normally with uniform sampling and no defense. It fixes clean quality and shows how badly attacks hurt.
The strongest clean-quality competitor is Liu et al.'s self-paced curriculum extractor. It reports 13.54 dB iSDR on Libri2talker, against 12.57 dB for random training. We re-implement it on our backbones and must reproduce roughly that gap before any comparison is trusted. Any defense we propose must stay within 0.5 dB of this model's clean iSDR to count as cheap on clean quality.
The strongest robustness competitor is whole-network adversarial training, run in two forms. One is uniform strong-attack training in the Madry style. The other is Cai et al.'s curriculum version, transplanted to extraction. For robust extraction there is no reported result found on Lacuna. For extraction with three or more interferers on Libri3Mix there is also no reported result found on Lacuna. Those numbers are therefore produced here.
Fairness is enforced by counting attack gradient steps in a shared GPU-hour budget. Every method gets the same data, epochs and hyperparameter search size, and is evaluated with the same attack suite. Our method beats whole-network training if it reaches equal or better worst-case SI-SDRi and redirection rate at each budget while losing less clean quality.
Several ablations isolate the two ingredients. Targeted adversarial training without the curriculum isolates the schedule. Curriculum adversarial training of a non-localized component, the mask estimator, isolates the targeting. Gaussian noise augmentation at matched energy tests whether the gain comes from worst-case examples rather than generic noise.
Why not just…?
Why not just plug in an adversarially robust speaker encoder? This is the most serious threat. If clue fragility lives entirely inside the encoder, a robust encoder fixes clue attacks with no new method. We run it as a baseline. The patching study decides the question directly: if encoder patching recovers most of the loss and fusion patching does not, we report that the cheaper fix suffices.
Even in that case, a robust encoder cannot defend against mixture attacks, which never pass through the encoder. Also, Wang et al. show that the conditioned layers are where clue information is lost. The fusion pathway is therefore a plausible failure site independent of embedding quality.
Why not just mix adversarial examples uniformly into training? In vision, Cai et al. found that uniform strong-attack training forgets weaker attacks and plateaus at 46.18% worst-case accuracy, against 69.27% with the curriculum. That evidence comes from classification, not extraction, so we do not assume it transfers. The no-curriculum ablation, evaluated across all budgets including weak ones, settles it for this setting.
Why not just quantize or denoise inputs? Input quantization helped Cai et al. However, preprocessing defenses can hide gradients and look robust only against naive attacks. We evaluate any such defense with adaptive attacks that differentiate through the preprocessing, plus multiple restarts. If quantization alone survives adaptive attacks at matched clean cost, it is the better answer and we will say so.
Proposed Experiments and Decisive Checks
The first experiment maps vulnerability. For each backbone, threat model and budget, we report untargeted SI-SDRi loss and redirection rate, broken down by interferer count and by target–interferer embedding similarity. Redirection rate is the fraction of outputs closer, in SI-SDR, to an interferer than to the target. We predict redirection rises with both interferer count and similarity.
The second experiment is the localization table of patching recovery fractions. The claim survives only if fusion-layer patching recovers more than half of the loss under clue attacks on both backbones. Under the same attacks we also track how strongly each layer's attention or conditioning weights respond to the clue, to test whether attacks re-create the inactive-weight decoupling.
The third experiment compares the defenses on the clean-versus-worst-case SI-SDRi trade-off. It uses Libri2talker, WSJ0-2mix and the one-to-three-interferer LibriMix sets, under matched compute.
The fourth is a transfer check. Attacks crafted on one backbone are applied to the other, mirroring the black-box transfer that Villalba et al. found for verification.
The last check uses REAL-M. Its blind SI-SNR estimator, which correlates 0.82–0.87 with true SI-SNR on synthetic data (REAL-M), tells us whether targeted hardening changes real-world quality. REAL-M is a separation benchmark rather than an extraction benchmark, so we use its mixtures with per-speaker enrollment segments and treat the result as indicative, not a headline.
References
- x-Vectors Meet Adversarial Attacks: Benchmarking Adversarial Robustness in Speaker Verification
- Neural Speaker Extraction with Speaker-Speech Cross-Attention Network
- Target Speaker Extraction with Curriculum Learning
- Curriculum Adversarial Training
- Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small
- Rethinking The Separation Layers In Speech Separation Networks
- LibriMix: An Open-Source Dataset for Generalizable Speech Separation
- Real-M: Towards Speech Separation on Real Mixtures