Speaker-Relative Emotion Conditioning for Expressive Text-to-Speech Trained on Doubly Skewed Data

Expressive text-to-speech systems learn emotion from recorded corpora, and those corpora are skewed in two ways at once. Most utterances are neutral. The few emotional ones are also unevenly spread across speakers' genders: one corpus may hold anger mostly from men and happiness mostly from women. A synthesizer trained on such data can quietly learn that "angry" partly means "sounds male." When asked for an angry female voice, it may then either drift toward a lower, male-like pitch profile or flatten the anger altogether.

We propose to measure this failure directly and to fix it in a way that keeps gender visible to the model rather than scrubbing it. The fix has two parts. First, we describe each emotion as a displacement: how far an utterance sits from that same speaker's neutral voice in a pretrained emotion embedding space. Second, during training we transplant displacements across speakers of different genders and check, with frozen pretrained judges, that the output carries the transplanted emotion while keeping the target speaker's identity.

Core Claim

Under compound emotion-by-gender skew, a single-stage text-to-waveform synthesizer conditioned on raw self-supervised emotion embeddings will show the largest emotion-fidelity losses in the rarest emotion-gender cells. Two changes should close most of that gap without hurting naturalness or the majority cells: making the conditioning speaker-relative, and adding a cross-gender transplant consistency loss. These changes should also beat both cell-balanced resampling and adversarial gender removal. The claim is falsifiable at the level of individual cells, such as "angry, female, held-out sentence," and not only on corpus averages.

Four lines of prior work motivate this design.

Emo-bias audits 15 self-supervised models fine-tuned for emotion recognition. It finds that downstream gender bias tracks the per-gender emotion distribution of the fine-tuning data (Pearson $r > 0.8$), while bias already present in the pretrained model barely predicts it. Fine-tuning on skewed data is therefore where bias enters. That paper studies recognition only and tests no mitigation. We move its diagnosis into synthesis and test a mitigation.

Twists, Humps, and Pebbles shows that in multilingual recognizers, higher internal gender extractability correlates with smaller gender error gaps ($\rho = -0.65$). That result argues against making representations gender-blind.

Gender De-Biasing in Speech Emotion Recognition found that adversarial gender removal cut overall concordance from 0.684 to 0.518 and trained unstably. That result constrains our design, so we keep gender explicit through a speaker embedding and remove only the speaker's baseline from the emotion signal.

Learning Emotional Representations from Imbalanced Speech Data addresses emotion scarcity with dual-level mixup and a consistency loss. It reports 60.29% unweighted average recall on IEMOCAP with 80% of non-neutral samples removed, and it uses the learned extractor as a reference encoder for an emotional TTS. We borrow its idea of a consistency objective across interpolated or transplanted signals. Our setting differs in three ways: that work trains its extractor from scratch, synthesizes through a cascaded spectrogram-plus-vocoder model, and never reports gender.

Research Program

The emotion representation is emotion2vec, kept frozen. It is a self-supervised model distilled from 262 hours of emotional speech, and a linear probe on it reaches 71.79% weighted accuracy on IEMOCAP. Its embeddings are known to retain speaker cues, and that residue is exactly the channel through which corpus skew turns into gender-emotion entanglement.

Let $\phi(x)$ be emotion2vec's utterance embedding of audio $x$, and let $\bar\phi_s$ be the mean embedding of speaker $s$'s neutral utterances, which are plentiful even in skewed data. The conditioning vector is the displacement $e = \phi(x) - \bar\phi_s$. Speaker identity enters separately through a speaker embedding, so gender stays fully extractable from the model's inputs.

The synthesizer is a VITS-style text-to-waveform model, as used in YourTTS. It is a single network that maps text straight to audio and is trained with variational and adversarial losses. Following JETS, it avoids the mismatch between a separate acoustic model and vocoder. The displacement $e$ conditions its duration predictor, flow, and decoder, in the way DC CoMix TTS conditions VITS on a style embedding while keeping a separate speaker table.

On each batch, alongside the usual reconstruction losses, we sample a displacement $e$ from speaker $s$ and pair it with a speaker $s'$ of the other gender. We then synthesize $\hat x = G(\text{text}, s', e)$ and apply

$$ \mathcal{L}_{\text{swap}} = \big\| \phi(\hat x) - \bar\phi_{s'} - e \big\|^2 \;-\; \lambda \cos\big(v(\hat x), v_{s'}\big), $$

where $v(\cdot)$ is a frozen speaker-verification embedding, $v_{s'}$ is speaker $s'$'s reference embedding, and $\lambda$ weights the identity term. In words, the transplanted emotion must show up as the same displacement from the new speaker's neutral voice, and the voice must remain that speaker's. The loss fills rare cells such as "female, angry" with gradient signal borrowed from the abundant "male, angry" cell. This differs from oversampling, which only repeats the few real examples.

This also differs from the Emotion Disentangling Module of Cross-Speaker Emotion Disentangling and Transfer. That module scrubs emotion from speaker embeddings with gradient reversal and orthogonality losses, inside a cascaded Tacotron2, on balanced acted data with a female-only target. Our method neither reverses gradients nor removes gender information.

Baselines

The naive reference point is the same VITS-style model trained on the skewed split with a learned embedding per emotion label and per speaker. It uses no self-supervised features and no balancing. It shows how much the skew hurts when nothing is done.

The strongest direct competitor is the same model conditioned on raw emotion2vec utterance embeddings instead of displacements, with the same reconstruction losses. This is the straightforward way to combine affective self-supervised features with end-to-end synthesis, and it isolates whether the speaker-relative transform matters.

The strongest existing imbalance method is the dual-level mixup Emotion Extractor of Wang et al.. We re-implement it as a reference encoder inside the identical VITS backbone, so that backbone differences do not confound the comparison. For orientation, that paper reports a naturalness MOS of 3.61 (baseline 3.55, ground truth 3.87) on ESD with 10 utterances per emotion per speaker, using RFTacotron. On end-to-end ESD synthesis with per-gender, per-emotion fidelity metrics, we found no reported result on Lacuna, so we will establish those numbers ourselves.

The adversarial competitor attaches a gradient-reversal gender classifier to the raw emotion2vec conditioning vector, as in Gorrostieta et al. and the Emotion Disentangling Module. Two ablations isolate our ingredients: displacement without the swap loss, and the swap loss applied to raw embeddings. An oracle model trained on the full, balanced ESD gives an upper reference.

All systems share the backbone, the data split, the number of training steps, and a tuning budget of an equal number of hyperparameter trials, each selected on a held-out validation set using worst-cell fidelity. Beating the field means two things on the decisive skewed split. First, the worst emotion-gender cell's emotion fidelity must improve over the raw-embedding baseline by more than the run-to-run spread across three seeds. Second, majority-cell fidelity and naturalness MOS must not fall measurably.

Why not just…?

Why not just resample so every emotion-gender cell is equally frequent? Duplicating the few female-angry clips teaches the model those particular clips. Vaidya et al. found that oversampling left disgust undetectable, and that it fits narrow "spikes" around individual speakers' quirks. Still, synthesis is not classification, and resampling is cheap. We therefore run cell-balanced resampling as a full baseline, both alone and combined with each conditioning variant. If resampling plus raw embeddings matches our method on held-out sentences and unseen-speaker transplants, our contribution shrinks to the benchmark, and we will say so.

Why not just remove gender from the emotion embedding adversarially? Gorrostieta et al. saw a large accuracy drop and instability from adversarial removal. Savoldi et al. found that gender-blind internals predict larger gaps. For synthesis there is an added cost: scrubbing gender from the conditioning may also scrub the gender-specific way an emotion is realized, such as typical pitch range. We still run the adversarial variant, because it is the textbook answer, and we compare its worst-cell fidelity and gender-recognizability of outputs against ours.

Why not just condition on emotion labels and skip self-supervised features? Labels carry no intensity or within-class variation, and they inherit the skew just as much. The naive baseline tests this directly. If labels plus resampling are as good as anything else, the self-supervised direction is not pulling its weight here.

Why not just subtract the speaker mean and stop there? This is the real threat, because the displacement transform is cheap and may already do most of the work. The displacement-only ablation settles it. The swap loss earns its place only if it improves the rare cells beyond what displacement alone achieves, especially for speakers who have zero recordings of a given emotion.

Proposed Experiments

The main testbed is the English portion of ESD, whose speakers and emotions are listed in ESD. Using speaker gender metadata, we construct three splits. The first removes 80% of non-neutral data uniformly, following Wang et al. The second applies the same removal and also skews the remainder by gender: anger kept mostly from male speakers, happiness mostly from female speakers, and some cells emptied entirely. The third adds held-out speakers whose emotional data is withheld, for zero-shot transplant tests.

Emotion fidelity of the synthesized audio is judged by an emotion classifier trained on the full balanced ESD, not by emotion2vec. This avoids rewarding the model for gaming its own training critic; Zhou et al. report 92% and 89% accuracy for such classifiers on ESD Chinese and English. A second judge, a WavLM-based classifier WavLM, checks agreement. Speaker similarity uses a verification model different from the one in the training loss. We also report per-cell fundamental-frequency statistics, the per-emotion gender gap $d_e$ defined in Emo-bias, worst-cell accuracy, naturalness MOS, and listener emotion-identification tests stratified by cell.

The decisive check is a controlled sweep of gender skew strength from balanced to fully one-sided within each emotion. Our prediction is that the raw-embedding baseline's worst-cell fidelity and its gender drift (target-female outputs shifting toward male pitch and timbre) worsen steadily as skew grows. Our method's should stay roughly flat. If both curves are flat, the entanglement hypothesis is wrong. In that case the fine-tuning-stage bias that Emo-bias found in recognition does not carry over to synthesis, which is itself a useful negative result.

A secondary check tests whether the method's outputs can repair recognition. We use the synthesized fill-in data to retrain an emotion recognizer on IEMOCAP and MSP-Podcast, then test whether it reduces the per-emotion gender gaps that Emo-bias documented, without the accuracy collapse seen under adversarial training.