Keeping Emotion When a Speech Synthesizer Learns New Voices

Suppose an expressive text-to-speech model can already speak in five voices, each in happy, sad, angry, surprised and neutral styles. Later it has to learn new voices. In practice those new voices usually arrive as plain, neutral recordings, because almost nobody records an hour of themselves acting angry. Fine-tuning on that neutral data is where the trouble starts. The model can go on sounding like the old speakers while quietly losing its ability to make them sound emotional. The new speakers never gain any emotional range, and emotion requests for them come out flat. Today's forgetting metric for speech synthesis checks only whether the voice still sounds like the same person, so it misses this entire failure.

This note proposes to measure that hidden, emotion-specific forgetting and to prevent it with a cheap, audio-free form of replay. The model separates speaking style (pitch, loudness and timing curves) from voice timbre. That separation lets the previous model act as a teacher for emotional style: it supplies emotional pitch and timing targets for the old speakers and for the new speakers, and neither requires storing any emotional audio. Speaker-emotion disentanglement is the mechanism in this proposal. Catastrophic forgetting, meaning a network overwriting old skills when trained on new data, is the setting it has to survive.

Core Claim

In a synthesizer that models prosody explicitly (prosody here meaning pitch, energy and duration), forgetting splits into two parts that behave differently. Timbre forgetting is the part prior work measured, and it is fixed by small audio replay. Emotion forgetting is invisible to speaker-verification metrics. It also cannot be fixed by audio replay alone, because the new speakers have no emotional audio to replay. We claim that distilling from the previous model in speaker-normalized prosody space fixes both halves of emotion forgetting at once. That distillation keeps old speakers' emotional contours intact and transfers them to new speakers, with a memory cost of zero emotional recordings.

This is different from the closest existing work. Continual Speaker Adaptation for Text-to-Speech Synthesis shows that experience replay (mixing a few stored old-speaker clips into each new training batch) keeps speaker identity in Tacotron 2 while weight regularization collapses. It studies neutral speech only, and it scores retention with a frozen speaker verifier that cannot see emotion. Continual Speech Learning with Fused Speech Features learns speaker identification and emotion recognition as separate classification tasks on separate corpora, so it never asks whether adding speakers damages emotion within the same signal. Towards Lifelong Learning of Multilingual Text-To-Speech Synthesis adds languages to a single neutral speaker. On the disentanglement side, DiEmo-TTS and the Speaker-agnostic Emotion Vector work treat emotion as a displacement relative to each speaker's neutral baseline, but both train offline on fixed data. We take that relative-displacement idea and turn it into a replay signal.

Research Program

The backbone is a FastSpeech 2 acoustic model. It has explicit pitch, energy and duration predictors, and it separates speaker from emotion conditioning in the prosody-versus-timbre style of iEmoTTS. Our method builds on this factorization. It does not propose a new disentangler.

The model is first trained on a base set of speakers with all emotions. New speakers then arrive one at a time with neutral data only. Timbre retention for old speakers uses the audio replay that Hemati & Borth showed works, including their buffer replication when memory is tiny.

Emotion is handled by cross-factor prosody distillation. For any text $x$ and emotion $e$, the frozen previous model predicts a prosody contour for each old speaker. We normalize that contour by the old speaker's neutral log-pitch mean and spread, which gives a speaker-free emotional shape $z_e(x)$ averaged over old speakers. For a new speaker $s$ we store only its neutral mean $\mu_s$ and spread $\sigma_s$, both computed from its own recordings. The prosody predictor is then trained with

$$\mathcal{L} = \mathcal{L}_{\text{new}} + \mathcal{L}_{\text{replay}} + \lambda \sum_{s,e} \big\| f_\theta(x, e, s) - (\mu_s + \sigma_s\, z_e(x)) \big\|^2 .$$

In this objective, $f_\theta$ is the current model's predicted log-pitch contour, with analogous terms for energy and duration. $\mathcal{L}_{\text{new}}$ is the ordinary loss on the new speaker's neutral audio. $\mathcal{L}_{\text{replay}}$ is the audio-replay loss for old speakers. $\lambda$ weights the distillation. The sum over $s$ covers both old and new speakers; for old speakers the term acts as Learning-without-Forgetting (a method that keeps outputs close to the previous model's) restricted to prosody. The texts $x$ can be any sentences, since no audio is needed.

Because the targets are low-dimensional contours produced by the previous model, the method avoids the error build-up that affects replay from generated audio in Deep Generative Replay. The model never regenerates spectrograms across rounds.

The analysis side asks whether factorization itself changes forgetting. We run the same sequence on an entangled latent-style model, a transformer TTS whose style embedding is disentangled from speaker only by a gradient-reversal layer (a layer that trains the embedding to be useless for predicting the speaker). We then compare where forgetting shows up in the two models.

Baselines

The do-nothing reference is sequential fine-tuning on each new speaker's neutral data with no protection. Its counterpart is a joint model retrained from scratch on all data seen so far, which serves as the upper bound. Both use the same backbone, the same steps per new speaker, and the same learning-rate search. Emotion transfer for new speakers comes out of joint retraining for free, because the model is trained offline on all speakers, so this model also marks the best achievable emotion rendering for unseen emotional combinations.

The strongest existing method is experience replay with buffer replication from Hemati & Borth. On VCTK with one stored clip per past speaker, it restores the first speaker's Retained Accuracy from 0.20 to 1.0. We run it with an audio buffer drawn from all emotions of old speakers. Its buffer size is matched to ours in stored seconds of audio, it gets the same tuning budget, and it runs on the same speaker order across three seeds. Our method uses the same buffer for timbre, so the comparison isolates what the distillation term adds. We count ourselves as beating it only if two things hold together. First, emotion accuracy on new speakers must improve by a margin that is larger than the spread across seeds. Second, old-speaker emotion accuracy must be at least as good as the replay baseline's, and speaker retention and word error rate must not get worse. No reported result was found on Lacuna for emotion retention under continual TTS, so this baseline number will be established here.

We also run an adapter-isolation baseline. It freezes the shared model, including the emotion pathway, and learns only a new speaker embedding and small adapter layers per speaker. Elastic Weight Consolidation (a penalty that anchors weights judged important for past tasks) is included for completeness, despite its reported collapse in TTS.

Two ablations isolate the key ingredient. The first drops speaker normalization and uses raw old-speaker contours as targets. The second keeps distillation for old speakers only and removes the new-speaker targets, which separates retention from transfer.

Why not just…?

Why not just use experience replay with emotional clips, which already works? Replay can plausibly keep old speakers' emotions, and we expect it to match us on that axis. The limitation is structural: the new speakers have no emotional audio, so replay cannot give them emotional range. Joint training on old emotional clips does offer implicit cross-speaker transfer. However, DiEmo-TTS and iEmoTTS both exist because implicit transfer tends to leak identity or drift toward neutral. If replay alone reaches emotion accuracy on new speakers within seed variance of our method, the transfer claim fails, and we will report that.

Why not just freeze everything and add a per-speaker adapter? This is the most serious threat. Frozen emotion weights cannot forget, and a well-disentangled model might render the new timbre with the old emotion pathway unchanged. Two concerns remain. Small adapters may fit timbre worse than full fine-tuning, and a frozen prosody predictor has never seen the new speaker's pitch range, which is the zero-shot drift problem listed in the research file. The experiment settles this directly. If adapters match on speaker similarity, emotion accuracy and pitch error, we conclude that good factorization makes emotion forgetting a non-problem. That result would be publishable in its own right, and we would recommend adapters over our loss.

Why not just apply standard output distillation to the whole spectrogram? Distilling full spectrograms ties timbre and prosody back together, and it gives no target for new-speaker emotions, because the old model cannot produce the new voice. The raw-contour ablation tests whether the normalization matters, which would show that the gain comes from disentanglement rather than from distillation in general.

Proposed Experiments

The main benchmark is the English half of ESD from Zhou et al.: 10 speakers, five emotions, 350 parallel utterances each. We train the base model on five speakers with all emotions and add the other five one at a time using neutral utterances only. The new speakers' emotional recordings are held out, and they are never used for training. This gives ground truth for measuring transfer. For a longer sequence of speakers, we then add VCTK speakers, which are neutral only.

Speaker retention is measured with a frozen verifier and embedding cosine similarity, following Hemati & Borth. Emotion is measured by running an emotion2vec-based classifier on the synthesized speech. That backbone is the model emotion2vec reports at 71.79% weighted accuracy on IEMOCAP, and we fine-tune it on ESD real speech. We report emotion accuracy separately for old speakers (retention) and new speakers (transfer). For new speakers we also measure pitch and energy correlation against their real held-out emotional recordings, since ESD's parallel design pairs each text with real emotional audio. Word error rate and a small listening test for naturalness and speaker similarity complete the evaluation. For scale, DiEmo-TTS reports 16.16% word error rate and a 3.80 naturalness score on Surprise in the offline setting.

The first decisive check tests whether hidden forgetting exists. If naive fine-tuning keeps speaker retention high while old-speaker emotion accuracy falls sharply, then the metric Hemati & Borth used misses a real failure. If both stay high, the premise is wrong.

The second check compares new-speaker emotion accuracy and pitch correlation across four systems: replay, adapters, our method, and joint retraining.

The third check tracks Surprise, which Emo-StarGAN found collapses into anger (0% on RAVDESS). We test whether Surprise is the first emotion to be forgotten, and we repeat the analysis on RAVDESS actors as out-of-domain listeners' targets. Finally, we compare how forgetting unfolds in the explicit-prosody model against the gradient-reversal model, to test whether factorization itself localizes damage.