Replaying Emotion Through the Voice, Not the Words

When a speech model learns new tasks one after another, it tends to forget how to recognize emotion. The usual fix stores real recordings from earlier tasks and mixes them back in. That is awkward for emotional speech, which is often private: crisis-line calls, therapy sessions, customer complaints. The alternative is generative replay: keep the ability to produce stand-in examples of past data instead of the data itself. The obvious way to do this with modern tools is to have a large language model (LLM) write emotional sentences and have a text-to-speech system read them. We argue that this obvious version is subtly wrong for emotion. If the LLM writes "I'm furious about this," the classifier can learn anger from the words and never needs the voice. The replay would then preserve a word detector while the model's sensitivity to prosody (pitch, energy, timing) quietly erodes.

We propose lexically decoupled, steered replay. For each past emotion the system keeps only a small activation steering vector, which is a direction in the hidden states of an expressive speech model that pushes its output toward that emotion. An LLM writes sentences that are emotionally neutral or that contradict the target emotion, and the speech model reads them aloud with the stored vector applied at several strengths. The replayed audio therefore carries emotion only in how it sounds, at graded intensities rather than as a single category. The continual learner has to keep its acoustic evidence for each emotion and the ordering of intensities within it. What gets stored per emotion is a vector, not recordings.

Core Claim

In continual speech learning where the audio encoder keeps being updated, replay that is synthesized in audio space, steered by stored emotion directions, and stripped of lexical emotion cues will retain acoustic emotion recognition better than three alternatives: replay of stored latent features, which goes stale as the encoder drifts; replay of LLM-written emotional text read by an instruction-prompted speech model, which teaches lexical shortcuts; and a small buffer of real audio at matched privacy cost. It will also preserve within-class intensity ordering, which discrete class rehearsal does not target at all.

Grounding in Prior Work

Three papers establish the gap. Continual Speech Learning with Fused Speech Features builds the only multi-task continual speech benchmark in the file. It chains keyword spotting, speaker ID, emotion recognition on IEMOCAP, intent, slot filling and ASR on Whisper. Its rehearsal stores raw audio, and it treats emotion as a four-way label. We keep its benchmark and backbone and replace stored audio with steered synthesis. On the text side, Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection generates past-class utterances from TF-IDF concept words and adds contrastive training and distillation. Class-Incremental Learning based on Label Generation expands label names into pseudo-replay sentences. Both assume the class lives in the words. That holds for intents and fails for vocal emotion. Our design deliberately removes that signal.

Two lines of work constrain the design. AlignCap shows that LLM-based speech emotion captioners, including SECap, hallucinate emotions implied by the transcript even when the voice is neutral. Lexical shortcuts are a documented failure in this domain, not a hypothetical one. An Interpretability-Guided Framework for Responsible Synthetic Data Generation in Emotional Text shows that keyword-guided LLM emotion data matches real-data expansion on tweets but collapses lexical diversity toward rigid adjective-noun patterns. Keyword-guided replay text would stack that narrowness on top of the lexical shortcut. We take the opposite route: diverse, affect-neutral or affect-contradicting text, with emotion supplied acoustically. Finally, Capacity-Gated Forgetting in LoRA Fine-Tuning finds that endogenous replay's 80% forgetting reduction at one seed shrinks to a +0.005 gain at seeds where no baseline forgetting occurred. Every claim here must therefore be reported across seeds and conditioned on whether forgetting actually happened.

Two papers supply the method. CoCoEmo computes emotion steering vectors as differences of mean activations between emotional and neutral speech in speech language models such as CosyVoice2. It injects linear combinations of these vectors in mid-to-late layers, and it reports that this controls emotion better than text instructions, including when the text contradicts the target emotion. That mismatch ability is exactly what lexically decoupled replay needs. Style Vectors for Steering Generative Large Language Models supports the claim that activation vectors are cheap to extract (at most 8 hours versus 150 for trained vectors) and scale continuously. EmoReg is a precedent for treating an emotion as a direction whose magnitude sets intensity. It also warns that pushing too far degrades intelligibility, so we cap the steering scale using a word-error-rate check. CoCoEmo and EmoReg use steering only at inference. They never ask whether a vector can stand in for a past task's data, and that repurposing is what we contribute.

Research Program

The learner is Whisper with the gated-fusion layer from the continual speech learning paper, trained on its six-task sequence. When the emotion task is active, we pass that task's training audio through a frozen expressive speech model, CosyVoice2 as used by CoCoEmo. We record mean activations per emotion class minus the neutral-class mean at the layers CoCoEmo identifies, and keep these vectors. IEMOCAP is not parallel, so class-mean differences may mix speaker and content effects. We will measure that by extracting vectors on the parallel ESD corpus as a control and comparing directions. The emotion audio is then discarded.

During each later task, a replay generator produces a mixed batch. An instruction-tuned LLM writes varied sentences under two constraints. Half are affect-neutral, checked by a text emotion classifier that must assign near-uniform probability. Half contradict the target emotion, for example cheerful content paired with the sadness vector. The speech model renders each sentence with the target emotion's vector at a scale $\alpha$ drawn from a grid between 0 and a cap $\alpha_{max}$. The cap is chosen as the largest scale at which Whisper's ASR word error rate on the synthesized audio stays within a fixed margin of unsteered speech. The replayed sample receives a soft target: the old model's posterior, used for distillation as in PAGeR, blended with the steered class according to $\alpha$.

The learner also gets a rank loss over pairs rendered from the same sentence at different scales. This loss asks that the predicted probability of the target class increase with $\alpha$, adapting the intra-class ranking idea in Fine-grained Emotional Control of Text-To-Speech from synthesis to recognition. Blended vectors, such as weighted sums of two emotions as in CoCoEmo, are replayed with matching mixed targets. This keeps the intermediate regions between discrete categories populated rather than letting them collapse.

Baselines

The naive reference is plain sequential fine-tuning through the six-task sequence with no protection. It shows how much emotion forgetting there is to fix, and we report it per seed.

The strongest existing method on this exact benchmark is the two-stage gated-fusion training with experience replay from Continual Speech Learning with Fused Speech Features. It reports 68.39% emotion recognition accuracy on IEMOCAP and 83.93% speaker ID accuracy within the sequence. We rerun it with our seeds and hardware, not just quote it. It stores real audio, so we run it at two buffer sizes. One matches the paper. The other matches our storage footprint, meaning bytes of steering vectors plus the reusable generator counted separately, which makes the privacy-storage trade-off explicit. Beating it means matching its emotion accuracy within seed variance with zero stored audio, or exceeding it at equal storage. We also include Learning without Forgetting and Dark Experience Replay++ as the paper does, with the same tuning budget.

For generative competitors, we run three. The first is feature-space Gaussian replay in the style of TRIPS ( Multivariate prototype representation for domain-generalized incremental learning): store per-class mean and covariance of fused encoder features and sample from them. The second is a PAGeR-style replay adapted to speech: LLM-written emotion-congruent sentences, keyword-guided, read by the same speech model given a text instruction ("say this angrily"), with PAGeR's contrastive and distillation losses. The third is our own pipeline with steering but emotion-congruent LLM text. It isolates lexical decoupling as the key ingredient. We also run a single-scale variant without the rank loss, which isolates the intensity component. All replay methods get identical numbers of replay samples per step, the same LLM and speech model where applicable, and the same hyperparameter search budget. We report generation wall-clock separately. The file gives no reported result for PAGeR or feature-Gaussian replay on speech emotion, so those numbers come only from our runs. The static IEMOCAP result of 87.5% accuracy from Learnable Fractional Superlets uses a different protocol. We cite it only as context, not as a target. Joint multi-task training serves as the upper bound.

Why not just…?

Why not just keep a small buffer of real recordings? It is simpler and has no synthesis artifacts, and at moderate buffer sizes it may well win on raw IEMOCAP accuracy. Our case rests on settings where audio cannot be retained. PAGeR reports that at a 1% memory budget generated pseudo-samples beat stored exemplars for text intents, but that has not been shown for speech. The matched-storage comparison above settles it. If real audio at equal bytes beats us, the contribution shrinks to the strict no-audio regime, and we will say so.

Why not just replay stored latent features, which is far cheaper than running a speech synthesizer? This is the most serious threat. If the encoder were frozen, Gaussian feature replay would probably be enough. Our claim depends on the encoder being updated by later tasks, which makes stored features stale. We test this directly. We run both methods with the encoder frozen and unfrozen, and we measure how far old-class feature means drift across tasks. If feature replay matches ours even with an updating encoder, audio-space synthesis is not worth its cost.

Why not just have an LLM write angry sentences and have the synthesizer read them angrily? This is the obvious combination. CoCoEmo reports that instruction prompts track human emotion rankings less well than steering, and AlignCap documents lexical hallucination. Neither shows the downstream effect on forgetting. The emotion-congruent-text baseline and the ESD acoustic-only probe below measure that effect. If the congruent baseline scores as well on ESD, our core mechanism is unnecessary.

Proposed Experiments

The main experiment runs the six-task sequence of the continual speech learning benchmark with five seeds. For each method we report emotion accuracy and F1 on IEMOCAP, average forgetting, and accuracy on the other five tasks, so that replay does not buy emotion retention at their expense. Following the LoRA-forgetting findings, we separate seeds by whether sequential fine-tuning forgot emotion. Mitigation is reported only where there was forgetting to mitigate.

The decisive check is an acoustic-only probe built from ESD. Because ESD speakers say the same sentences in all five emotions, words carry no label information. After the sequence ends, we evaluate each method's frozen model on held-out ESD speakers for the four classes shared with IEMOCAP. If lexically decoupled replay helps as claimed, its advantage over the congruent-text and instruction-prompt baselines should be larger on ESD than on IEMOCAP, where transcripts leak emotion. If the gap does not appear, the lexical-shortcut hypothesis is wrong.

A second check measures graded affect. We use IEMOCAP multi-rater annotations as soft ground truth, following CoCoEmo, and report rank correlation between the model's class probabilities and rater vote shares. Separately, on held-out synthesized sentences at graded steering scales, we report whether predicted probability increases with $\alpha$. This test is circular, so we also require the correlation on real multi-rater data.

A third check is a domain-shift sequence: ESD, then IEMOCAP, then EMO-DB. Here emotion stays the task and only the recording condition changes. This asks whether steering vectors extracted on English acted speech still produce useful replay after the learner has adapted to German. For every synthesized replay set we report intelligibility as Whisper word error rate and generation cost, so that gains are not explained by degraded audio or by unequal compute.