Adversaries That Must Generalize: Open-Set Speaker Removal from Speech Representations

Many speech systems try to strip "who is speaking" out of a representation while keeping "what is said." Voice conversion does this so a new voice can be swapped in. Privacy-preserving speech recognition does it so transcripts can be computed without exposing a biometric. The standard tool is an adversary: a speaker classifier trained on the representation, whose gradients are flipped so the encoder learns to fool it. The mechanism is a gradient reversal layer, a layer that passes features forward unchanged but multiplies gradients by a negative constant on the way back. The evidence that this works is weak in a specific way. Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion? drove closed-set identification of 251 training speakers from 93.1% down to 2.5%. Yet an x-vector verifier trained on the resulting features reached an equal error rate (EER, the operating point where false accepts equal false rejects; 50% is chance) of 19.56% on unseen speakers, lower than the 23.07% for the non-adversarial baseline. The encoder had learned to dodge one classifier's fixed decision boundaries, not to remove speaker identity.

We propose treating speaker removal as a domain generalization problem, with each speaker acting as a domain. The adversary is a pairwise verifier that answers "same speaker or not?" rather than a closed-set classifier. It is trained and applied in episodes where the speakers it scores are disjoint from the speakers whose classification boundaries the encoder has just been pushed against. This mirrors the meta-train/meta-test splits of domain-generalization work. The encoder is rewarded only for hiding identity in a way that transfers to speakers the adversary has not memorized. Borrowing from private/shared encoders in domain adaptation, we also give speaker information an explicit private channel, so the content encoder is not pushed to hide identity by distorting phonetics. The claim is testable: under a fixed word error rate budget, disjoint-speaker verification adversaries should leave the open-set attacker's EER far closer to chance than closed-set gradient reversal does.

Core Claim

The failure reported by Srivastava et al. is a generalization failure of the adversary, and it can be fixed by changing what the adversary is asked to generalize over. A closed-set speaker classifier defines a 251-way partition. An encoder can drive its accuracy to near zero by moving features across that partition's boundaries while preserving, or even sharpening, the within-speaker clustering that a metric verifier exploits. A verification adversary scored on held-out speakers cannot be beaten this way, because the only strategy that fools it on unseen speakers is to shrink the gap between same-speaker and different-speaker distances in general. What is new is the combination of two things: the adversary is open-set and pairwise, and it is trained with a speaker-disjoint episode schedule.

Relation to Prior Work

Two lines of work motivate the proposal. The first uses closed-set adversaries to remove speaker identity from content. Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations adds a gradient-reversed speaker classifier to an autoencoder and reports an external verifier's accuracy on the content code falling from 91.6% to 45.1%. That verifier was evaluated on the same 20 training speakers, so it is exactly the closed-set measurement the privacy paper warns about. W2VC applies gradient-reversal distillation to WavLM features and adds a phoneme-level CTC loss (connectionist temporal classification, which aligns frames to phoneme labels) to protect content. It reaches 1.63% WER on converted speech and a speaker similarity MOS of 3.62. We keep W2VC's CTC anchor, because it addresses the separate risk of destroying vowel formants. We replace its adversary.

The second line, the negative result from Srivastava et al., constrains the design. It rules out any evaluation that reports only closed-set accuracy, and it rules out any method whose adversary sees a fixed speaker set. Our evaluation protocol is built around its open-set attacker.

Domain adaptation work supplies the separation machinery. Genre Separation Network splits text into shared and private encoders with an orthogonality penalty and a reconstruction loss. It beats global adversarial alignment (CNN+DANN) by 1.2 to 1.7 F1, which it attributes to domain idiosyncrasies polluting the shared space. Enhanced Separable Disentanglement adds a loss that makes the private features useless for the task and a reconstruction loop. We reuse the private branch and reconstruction so speaker information has a legitimate place to go.

SCD-Conformer works in the opposite direction, removing content from speaker embeddings with Pearson decorrelation. Its linear penalty is a useful weaker comparator for our separation term. VDANN shows that plain domain-adversarial training can hurt verification: 11.62% versus 11.30% EER on SRE16, because it warps embedding geometry. That warns us that adversarial pressure has side effects, and it is part of why we monitor WER and reconstruction quality rather than privacy alone.

Research Program

The model has three parts. A content encoder maps filterbank or WavLM features to frame-level codes $z_c$. A private speaker encoder maps the utterance to a vector $z_s$. A decoder reconstructs the input from $(z_c, z_s)$. The content code is trained with CTC (and, for recognition, the full ASR loss). The speaker vector is trained with a speaker loss. The adversary $A$ takes two pooled content codes and outputs a same-speaker probability. Training runs in episodes. The training speakers are split each episode into a pool $P_{\text{fit}}$, on which $A$ is updated, and a disjoint pool $P_{\text{hold}}$, on which the encoder's adversarial loss is computed. Pools rotate so every speaker plays both roles over training. The encoder objective is

$$ \min_{E}\; L_{\text{task}} + \gamma L_{\text{rec}} - \lambda\, L_{A}\big(A;\, P_{\text{hold}}\big), $$

where $L_{\text{task}}$ is the CTC or ASR loss, $L_{\text{rec}}$ is the reconstruction error from $(z_c, z_s)$, $L_A$ is the verifier's binary cross-entropy on held-out-pool pairs, and $\gamma, \lambda$ are weights. The adversary itself minimizes $L_A$ on $P_{\text{fit}}$. Because $A$ is always scored on speakers it was not just fitted to, the encoder only gains from changes that defeat a generalizing verifier.

The analysis part asks why closed-set adversaries leak. On the Srivastava setup, we will measure how within-speaker and between-speaker distances of pooled content codes evolve under each adversary. Our hypothesis is that closed-set reversal compresses the classifier's logit margins while within-speaker variance falls. That would explain why the representation became easier for a metric attacker. If the distance statistics do not show this, the mechanism story is wrong even if the method works, and we will report that.

Baselines

The naive reference is the ASR model trained with no adversary, following Srivastava et al.'s ESPnet CTC/attention setup on LibriSpeech. It gives the floor for WER and the attacker EER that any privacy method must improve on; the reported value is 23.07% EER.

The strongest existing method on this exact question is closed-set gradient-reversal training as in Srivastava et al.. At its strongest setting ($\alpha=2.0$) it reports 2.5% closed-set accuracy, 19.56% open-set EER, and WER rising from 10.9% to 12.5%. We re-run it in our codebase with the same encoder, data, and number of updates. Its reversal weight gets the same tuning budget as our $\lambda$: an equal number of sweep points, each selected by the same held-out attacker. For voice conversion, the strongest competitor is W2VC on VCTK. It reports MCD 8.901, naturalness MOS 4.45, similarity MOS 3.62, and WER 1.63%. We retrain it with our adversary swapped in and all else fixed. Beating the privacy baseline means an open-set attacker EER clearly above 23.07%, the non-adversarial level, at WER no worse than 12.5%. The ambitious target is EER within a few points of 50%. Beating W2VC means matching its WER and naturalness while raising target-speaker similarity, since less source-timbre leakage should help the decoder follow the target embedding.

Three ablations isolate the ingredients. The first uses the same pairwise verifier trained on the same speakers the encoder is pushed against, which isolates speaker-disjoint episodes. The second removes the private branch and reconstruction, which isolates the separation machinery. The third replaces the private branch's separation with SCD-Conformer-style Pearson decorrelation, which tests whether a linear penalty suffices.

Why not just…?

Why not just use more speakers in the closed-set classifier? A larger partition is harder to dodge, and Srivastava et al. used only 251 speakers. This is a real alternative. We will run closed-set reversal on all LibriSpeech training speakers under the same tuning budget. If its open-set EER matches ours, then the fix is scale, not episodes, and the note's claim fails. We expect it to help partially but still be dodgeable, because any fixed class set leaves a boundary to move across.

Why not just attack a frozen, pretrained x-vector verifier? This is metric-based and open-set, so it removes much of our novelty on paper. The weakness is that a frozen network is a fixed target, and encoders can learn perturbations specific to it, the same way the closed-set classifier was dodged. Our evaluation always retrains a fresh attacker on the purified features, which exposes this. The frozen-verifier adversary is included as a baseline. If it holds up under a fresh attacker, we would adopt it and credit it as the simpler solution.

Why not just perturb the signal instead of training an adversary? End-to-End Voice Conversion with Information Perturbation removes speaker cues with formant shifting, pitch randomization, and random equalization. It reports 12.89% WER on VCTK against W2VC's 1.63%, which suggests a content cost. Perturbation also offers no guarantee about what an attacker can recover. We include it on both benchmarks, measured by the same fresh open-set attacker.

Proposed Experiments

The decisive experiment replicates the Srivastava protocol on LibriSpeech. For every method, we train fresh x-vector attackers of increasing capacity on the frozen content representation of unseen speakers, and we plot attacker EER against capacity alongside WER. The proposal succeeds if disjoint-speaker verification adversaries keep EER high across attacker capacities while closed-set reversal falls below the non-adversarial 23.07%. Srivastava et al. already showed that fall at one capacity; the curve tests whether it holds as attackers grow.

The second experiment is voice conversion on VCTK with unseen speakers held out, reporting MCD, WER, naturalness and similarity MOS, and the same fresh-attacker EER on the content code.

The third is a stress test on 3D-Speaker. Here the attacker is trained on one device or distance and tested on another, to check whether removal survives acoustic domain shift. There is no reported result found on Lacuna for adversarial speaker removal on that corpus. We will report its supervised verification baselines (6.55% cross-device, 9.45% cross-distance EER) only as context for attacker strength.

A negative outcome would also be informative. If no adversary pushes a fresh attacker near chance without WER exceeding 12.5%, that bounds how separable speaker identity and content are in these features.