Stratified Adversaries for Speaker-Free Prosody Transfer

Expressive speech synthesis tries to take the speaking style of one recording (its pitch movement, rhythm, emotion) and render it in a different person's voice. The standard way to stop the reference speaker's voice from leaking into the output is an adversary. A small classifier tries to guess who is speaking from the style representation, and the style encoder is trained to defeat it. This works, but it comes at a known cost. Push the adversary hard and the transferred speech goes flat and neutral. We argue that much of this cost has a simple cause. In real training pools, speaker and style are confounded: one corpus is a calm newsreader, another is an excitable podcaster. Any style information is therefore also evidence about who is speaking, and an adversary that removes everything predictive of the speaker must remove style too.

We propose to change the question the adversary asks. It should not ask "can you tell the speaker from this style code?" but "can you tell the speaker apart from other speakers who share the same style domain?" Concretely, we use a shared synthesis backbone with lightweight per-domain style experts, a small router network that softly assigns each reference clip to those experts, and one speaker adversary per expert. Each adversary only discriminates among speakers inside its domain. The experts keep domain-specific expressive detail, and the adversaries remove only the speaker information that the style domain does not already explain. The prediction is testable: on a corpus where we can dial the speaker–style confound up and down, the flattening that comes with stronger adversaries should mostly disappear.

Core Claim

The monolithic adversary used in Daft-Exprt, which attaches a speaker classifier to the prosody encoder through a gradient reversal layer (a layer that passes features forward unchanged but flips the sign of the classifier's gradient), minimizes the mutual information $I(z;s)$ between the style code $z$ and the speaker $s$. We propose to minimize

$$I(z; s \mid d) = I(z;s) - I(z;d) + I(z;d \mid s),$$

where $d$ is the style/recording domain, either a corpus or emotion label or a soft router assignment. The identity shows what changes. The marginal objective also penalizes the part of $z$ that encodes the domain, $I(z;d)$, whenever domains differ in speaker mix. The conditional objective does not. Suppose style were fully determined by domain and every domain contained several speakers. Then a style code could be perfectly expressive while carrying zero conditional speaker information. Under the marginal objective, the same code would be penalized.

Two honest limits follow from this. First, conditional invariance is only identifiable where domains contain more than one speaker. If every speaker appears in exactly one style, conditioning on $d$ makes the speaker constant, and the adversary goes silent. Second, the claim is falsifiable. If the conditional adversary sits on the same expressiveness–leakage curve as a well-tuned marginal one, the confound is not the cause of flattening.

Research Program

The backbone follows the separation that Multi-Speaker Multi-Style Speech Synthesis with Timbre and Style Disentanglement found effective. That work feeds speaker identity only to the FastSpeech2 decoder and style identity only to the variance adaptor (the module predicting pitch, energy and duration), with utterance-level pitch and energy normalization. We keep that wiring and that normalization. We replace their discrete style IDs with a reference encoder, so unseen styles and speakers can be handled. Duration is predicted, not copied, which avoids the fixed-length limitation of StyleVC.

The multi-expert part borrows from vision domain generalization. We add $K$ small style adapters on the variance adaptor, one per domain. A router outputs mixing weights $w$ over them, blended with a uniform average by a safety parameter $\alpha$. This is exactly the fusion rule of Best Sources Forward, which trained source-specific heads plus a domain-predicting router and found the router interpolating between neighboring experts on unseen domains. Their work was classification on fixed domain labels. Ours uses the router weights a second time, to weight per-expert speaker adversaries. This mirrors how MFSAN aligns distributions within domain-specific feature spaces rather than globally.

When domain labels exist (ESD emotions, Mandarin corpus identity), the router is supervised. When they do not, $d$ is obtained by clustering reference-encoder statistics. Whether discovered domains suffice is an explicit question; the file names latent domain discovery as an open problem for multi-expert models.

We also measure leakage directly rather than only through listening tests. We train fresh post-hoc probes to predict the speaker from frozen style codes, both marginally and within domain. We also probe emotion from the same codes. This lets us read off where each method sits on the leakage–expressiveness plane.

Baselines

The naive reference point is the same backbone with a reference encoder and no adversary. It shows how much leakage exists before any scrubbing.

The strongest existing method on this question is marginal adversarial scrubbing in the style of Daft-Exprt, reimplemented on our backbone. On its own data, Daft-Exprt reports MUSHRA 58.0 against roughly 36–41 for GST-Tacotron, VAE-Tacotron and Flowtron, pitch correlation (PCC) 0.43 against roughly 0.07–0.14, and naturalness MOS 3.28. Those figures come from an internal 12-speaker corpus and are not comparable to ours, so we rerun the method. Fairness is ensured by identical data, steps and compute. We sweep the reversal weight $\lambda$ over the same grid we give our method and report its whole frontier, not one point. Beating it means a frontier that strictly dominates: lower within-domain speaker-probe accuracy at equal or higher emotion-classifier accuracy and pitch correlation.

The second strong competitor is the static separation of Song et al., run with its utterance normalization. On its Mandarin corpora it reports speaker similarity MOS between 4.23 and 4.66 and style similarity MOS between 4.08 and 4.62. On noisy podcast data, normalization at the utterance level scored 3.91 against 3.82 for speaker-level normalization. We rerun it on the same Mandarin pool for seen speakers and styles.

On ESD, the best reported results we found are from RSET: MOS 3.98 against 3.89 for EmoMix, and similarity MOS 4.15. It uses mutual-information minimization via an upper bound (vCLUB) rather than an adversary. We include an unconditional vCLUB variant on our backbone as a non-adversarial marginal baseline.

Three ablations isolate the ingredients. The first is a conditional adversary with no experts: a single style head, with the discriminator given $d$ as input. The second is experts with a marginal adversary. The third uses hard instead of router-weighted stratification.

Why not just…?

Why not just tune the adversarial weight on a single shared model? Daft-Exprt already documents that raising the weight trades expressiveness for invariance. Our analysis predicts the trade-off is structural under confounding, not a tuning artifact. The $\lambda$ sweep in our baselines tests this directly. If some $\lambda$ reaches our point on the plane, the proposal fails.

Why not just condition one discriminator on the style label, without any experts? This is the real threat. It implements the same conditional objective with far less machinery, and it is our first ablation. Our hypothesis for why experts still matter is twofold. A single shared style head averages domain-specific prosody across domains, the seesaw effect. And the router is needed to supply $d$ for unlabeled references at test time. If the expert-free conditional adversary matches the full system on seen and unseen references, we will report the conditional objective as the contribution and drop the experts.

Why not just use a temporal bottleneck such as word-level prosody latents? CopyCat and eCat restrict prosody codes to word-level windows so that utterance-global timbre cannot fit. eCat reports closing 46.7% of the naturalness gap between CopyCat2 and human recordings. But a bottleneck limits capacity regardless of whether information is speaker or style, and word-level pitch range still correlates with speaker. We run a word-level bottleneck variant without adversary and probe it the same way. If within-domain speaker probing is already at chance there, the adversary is unnecessary.

Why not just normalize pitch and energy? Utterance normalization is already in every system we compare, including ours, so its contribution is held constant. It touches only scalar contours, while timbre also lives in the spectral envelope. Any residual leakage above the normalization-only naive baseline is what the adversaries must remove.

Proposed Experiments

The decisive experiment is a controlled confounding sweep on ESD, described in Emotional Voice Conversion: Theory, Databases and ESD. ESD has 20 speakers, each recording the same 350 sentences in five emotions. We construct training subsets ranging from fully crossed, where every speaker has every emotion, through partially disjoint, to nearly disjoint, where each emotion is recorded by only two or three speakers. We stop short of one speaker per emotion, where our objective is unidentifiable; we also run that endpoint and expect our method to fail there.

The prediction is specific. The marginal adversary's frontier should degrade as confounding grows. The stratified one should stay near its fully-crossed frontier until the identifiability limit.

Metrics are computed on cross-speaker transfer with held-out speaker–emotion pairs. Emotion fidelity is measured with an emotion classifier trained on real ESD; the ESD paper reports 92% accuracy on Chinese and 89% on English for emotion recognition, which serves as a ceiling. We also measure pitch correlation with the reference, speaker-verification similarity to the target, and within-domain and marginal speaker-probe accuracy on style codes. Intelligibility is tracked by character error rate.

The second experiment is the six-corpus Mandarin pool from Song et al. There, corpus identity confounds speaker, style and recording channel, including noisy podcast speech. This tests whether corpus-level experts plus within-corpus adversaries also absorb channel differences, and whether router-discovered domains match labeled ones.

Zero-shot evaluation uses held-out ESD speakers as references, mirroring the unseen-reference setting Daft-Exprt probes with RAVDESS. Listening tests (MUSHRA for prosody transfer, MOS for naturalness, ABX for speaker similarity) follow on the frontier points chosen by objective metrics.

The study counts as successful only if three things hold. Stratification must dominate the swept marginal frontier under heavy confounding. The gap must shrink in the fully crossed condition, which is the internal control. And the expert-free conditional ablation must be compared honestly against the full system.