Text-to-speech systems have to choose a melody, rhythm and loudness for every sentence, and the same words can be spoken in many valid ways. The best current prosody predictors handle this with normalizing flows: invertible networks that map each realized pitch, energy and duration pattern to a point in a simple Gaussian "latent" space and back again. These models are trained once and then frozen. A narrator who drifts into a slower, lower register over a long chapter, or a conversational partner whose pitch range the system should gradually match, cannot change the model's behavior except through a fixed window of preceding text. Updating the network on the fly brings back the familiar problem of continual learning: it absorbs the new speaker and loses the voice it started with.
We propose to resolve this the way complementary learning systems theory says the brain does. One system learns slowly and preserves structure; a separate fast system stores recent experience. The fit with flows is close. Because a flow is exactly invertible, every utterance the system hears or produces can be mapped losslessly into its latent space. A fast memory can then update only the base Gaussian, its mean and scale, from those latents in closed form, while the invertible transform, which holds all the slow knowledge of how text maps to prosody, stays untouched during a session. Adaptation is immediate and costs no gradient steps. Within a session it cannot cause forgetting, because no weights move. A separate, explicitly controlled consolidation phase decides what part of the recent experience deserves to change the slow network.
Core Claim
In a conditional flow prosody model, the right place for fast plasticity is the base distribution and the right place for slow plasticity is the transform. A small shift of the base mean in latent space corresponds, after the flow, to a text-dependent and nonlinear change in the predicted contour. It is therefore more expressive than shifting a speaker's average pitch or speaking rate, and much cheaper and safer than fine-tuning. Concretely, the fast system keeps a discounted posterior over the base mean for each latent dimension:
$$ \mu_t = \frac{\kappa_0 \mu_0 + \sum_{i \le t} \lambda^{t-i} z_i}{\kappa_0 + \sum_{i \le t} \lambda^{t-i}}, $$
Here $z_i$ is the latent obtained by inverting the flow on the $i$-th observed utterance given its text, $\mu_0 = 0$ is the prior mean learned in training, $\kappa_0$ is the prior's strength in pseudo-observations, and $\lambda \in (0,1]$ is a forgetting factor that lets the memory track drift. An analogous update applies to the per-dimension scale. The slow flow weights change only during offline consolidation. In that phase we replay stored latents through the flow, together with their text, as pseudo-rehearsal, and we keep a fast and a slow exponential-moving-average copy of the flow. For each replayed sample, the copy that assigns the higher log-likelihood serves as the distillation target. This is the continuous-density analogue of the confidence-based teacher selection in CLS-ER, which picks between its plastic and stable models by their softmax confidence on the true class. The research file found no published work combining these two lines.
Relation to Prior Work
The flow side supplies the method. VarianceFlow replaces FastSpeech 2's mean-squared-error pitch and energy predictors with rational-quadratic spline flows. It reaches MOS 4.19 on LJSpeech against 3.92 for FastSpeech 2, and its invertibility is what lets us compute exact latents and exact likelihoods. eCat predicts word-level prosody latents with a Glow flow conditioned on a 72–95 word RoBERTa window, and reports those latents to be largely disentangled from speaker timbre. That property makes it plausible that latents from a partner's speech carry transferable style. Both models are frozen after offline training. We keep their predictors and add a fast and a slow learner around them.
The conditioning-prior work motivates the design. CUC-VAE shows that an utterance-specific prior computed from neighboring sentences beats unconditioned Gaussian sampling, with WER 9.9% against 14.8% on LJSpeech. LCPVAE shows that a learned speaker-conditioned prior avoids over-regularizing the latent space. Both priors are fixed functions of a closed training set. Our fast system is a prior that keeps updating from realized speech rather than from text alone.
The continual-learning side constrains the design. CLS-ER's three full model copies and raw replay buffer are listed as limitations in Learning Fast, Learning Slow. We confine the per-utterance fast system to a few hundred numbers of sufficient statistics and store latents instead of audio. Continual-MAML adapts fast weights by gradient steps triggered by spikes in the loss. Our fast step is gradient-free, which removes that step-size and trigger tuning from the session loop. The forgetting factor $\lambda$ still has to be set, and we sweep it.
Research Program
We first build a flow prosody predictor at phoneme level in the style of VarianceFlow, modeling log-F0, energy and log-duration with a CUC-style text context encoder. We train it on the LibriTTS training speakers. We then verify that latents of held-out speakers are systematically off-center: per-speaker mean latents should be significantly non-zero and should differ consistently between speakers. If they are not, the base-space hypothesis has nothing to act on, and we will know immediately.
Second, we implement the fast base-posterior update above and study three things: how adaptation quality depends on the amount of observed history, on $\lambda$, and on which latent coordinates are adapted. Third, we implement consolidation and measure how well it retains performance on earlier speakers when sessions arrive in sequence. Fourth, on DailyTalk, we test a two-source fast memory that combines the agent's own previous turns with the partner's turns mapped through the agent's flow, as a mechanism for prosodic entrainment. Throughout, the flow's exact negative log-likelihood (NLL) on held-out prosody serves as a cheap, sensitive objective metric before any listening test is run.
Baselines
The naive reference is the frozen flow predictor sampling from the unadapted standard base, with the same text-context window, applied to every utterance of the stream. Every adaptive method must beat it on streaming prosody NLL to count as adapting at all. As a published anchor for synthesis quality on these corpora, CUC-VAE reports LibriTTS MOS 3.63 and MCD 6.04 dB (Cross-Utterance Conditioned VAE for Speech Generation). We will re-run it under our vocoder so that our MOS comparison is like-for-like, without importing that number. For streaming or continual prosody adaptation, we found no reported result on Lacuna, so every adaptive comparison is internal.
The strongest competitor is an in-context acoustic history encoder. This is the DailyTalk-style context encoder (DailyTalk, whose conversational FastSpeech 2 reaches pairwise MOS 3.90 against 3.86), extended so that it attends over the realized prosody of previous utterances as well as their text, and feeds the result into the same flow. It is amortized and needs no test-time computation, and it could learn entrainment directly from data. It is trained on the same LibriTTS and DailyTalk splits, with matched parameter count and the same training compute, and its history window is tuned on validation data. To beat it, our method must achieve lower NLL and comparable or better MOS at equal history length, and pull clearly ahead as history grows beyond the encoder's trained window or as speakers move away from the training distribution.
The second strong competitor is test-time fine-tuning. We run two variants on the same stream. The first fine-tunes low-rank adapters on the flow while the base network stays frozen. The second fine-tunes the full flow with reservoir experience replay. Each gets a tuning budget for learning rate and step count equal to the grid we spend on $\lambda$ and $\kappa_0$.
Two ablations isolate the key ingredient. The first adapts per-speaker mean and variance of F0, energy and duration in raw prosody space, which tests whether the latent space adds anything. The second gives the same parameter budget to the last coupling layer instead of the base, which tests whether the base is the right place for fast plasticity. For consolidation, the ablation replaces likelihood-selected teacher choice with a single EMA teacher.
Why not just…?
Why not just give the context encoder the acoustic history? This is the real threat, and if it matches us at all history lengths the idea should be dropped. It has two structural disadvantages. Its memory is bounded by the trained window, and DailyTalk observes that conversational gains only become pronounced after the fifth turn. Its behavior on unseen speakers is also limited by the training speakers, whereas the base posterior is a statistic that grows without limit and is consistent by construction. The experiment that settles it plots NLL against history length on unseen LibriTTS speakers and on the later turns of DailyTalk. Whichever curve keeps improving wins.
Why not just normalize pitch and rate per speaker? Raw-space normalization captures register and tempo but not text-dependent habits. Examples are a speaker who lengthens phrase-final words, or one who reserves wide pitch excursions for contrastive stress. The raw-space ablation measures this directly. If it closes most of the gap, the latent-space claim is false and we will report that.
Why not just fine-tune small adapters per speaker? Adapters avoid forgetting by isolating parameters. They still need gradient steps, a stopping rule and per-speaker storage, and they are noisy when only a handful of utterances have been heard. Evidence from classification that small-buffer replay methods behave poorly (DER++ 10.96% against CLS-ER 23.47% on S-Tiny-ImageNet, from CLS-ER) does not transfer automatically to prosody. We therefore measure adaptation quality per number of utterances observed, especially in the 1–10 utterance regime where a closed-form conjugate update should be most data-efficient.
Proposed Experiments
The main experiment streams the utterances of each held-out LibriTTS speaker in chapter order. Before each utterance, the model predicts its prosody from everything observed earlier. We report exact prosody NLL, F0 RMSE, duration error, and ASR WER after synthesis. MOS and pairwise preference tests are run on a fixed subset, using the same vocoder for all systems.
The continual experiment passes sessions from K speakers through consolidation in sequence, then measures NLL on the training speakers and on the first-seen speakers. This yields a forgetting curve for consolidation with and without likelihood-selected teachers, compared against full fine-tuning with replay. LJSpeech serves as an untouched held-out check of the original voice.
The entrainment experiment on DailyTalk compares own-history, partner-history and combined fast memories against the acoustic context encoder, scored by turn index, with listening tests targeted at turns six and later. The decisive outcomes are three. First, base-space adaptation must beat raw-space normalization and last-layer adaptation at equal budget. Second, it must overtake the acoustic context encoder once history exceeds the encoder's window. Third, consolidation must retain original-speaker NLL within noise of the frozen model while keeping part of the adaptation gain. If the first fails, the central claim is wrong. If only the third fails, the fast system is still a usable result without consolidation.
References
- Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System
- Varianceflow: High-Quality and Controllable Text-to-Speech using Variance Information via Normalizing Flow
- eCat: An End-to-End Model for Multi-Speaker TTS & Many-to-Many Fine-Grained Prosody Transfer
- Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speech
- A Learned Conditional Prior for the VAE Acoustic Space of a TTS System
- Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning.
- Cross-Utterance Conditioned VAE for Speech Generation
- DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech