Pre-Training a Speaker-Free Recurrent Prosody State for Long Conversations

Speech synthesizers that voice both sides of a conversation have to do two things at once. They must keep each speaker sounding like themselves, and they must let the manner of speaking drift naturally as the dialogue unfolds: speakers get more excited, slow down, echo each other's energy. Current conversational text-to-speech systems learn this second skill from tiny dialogue corpora. The standard benchmark, DailyTalk, has only two speakers. These systems also look back only a few turns. This proposal pre-trains a small recurrent network, a model that carries a running memory vector from one step to the next, on thousands of speakers' consecutive sentences and dialogue turns. The network is trained to track how prosody evolves across utterances: prosody here means rhythm, pitch, pausing and energy. Speaker identity is deliberately kept out of that memory and handled by a separate, tiny set of per-speaker parameters.

The bet is that "how prosody moves across a discourse" is a speaker-general skill. If so, it can be learned from large multi-speaker data the same way pre-training already supplies text-to-rhythm knowledge for single sentences. It should then transfer to a small dialogue corpus and hold up in the late turns where existing systems weaken.

Core Claim

Multi-speaker pre-training is known to transfer sentence-internal prosody. A study on the efficacy of model pre-training in developing neural text-to-speech system pre-trained on about 1,000 hours of LibriSpeech. It found that the gain lies mainly in "tone and rhythm" (CMOS +0.157) rather than pronunciation (−0.041), and argued that pre-training learns text-derived prosody.

Conversational work argues this is not enough. H4C-TTS models recent and older turns with separate encoders because text-only prosody sounds plausible in isolation but disconnected from the dialogue. Our claim is that the missing piece is a pre-trained cross-utterance prosodic state. It is a gated recurrent unit (GRU), a recurrent cell with learned gates that decide what to keep and what to overwrite. The GRU is updated once per utterance from that utterance's text embedding and measured prosody. We make two claims about it. First, when this state is pre-trained on many speakers and then adapted, it improves late-turn prosody on DailyTalk more than the same state trained only on DailyTalk. Second, it does so without degrading speaker similarity, because identity lives elsewhere.

The recurrence is not the whole claim. The experiments are built so that an attention-based memory given the identical pre-training can refute the recurrent part on its own.

Research Program

The backbone is FastSpeech 2, a non-autoregressive synthesizer that predicts duration, pitch and energy per phoneme and then generates spectrogram frames. Speaker identity enters only through the conditional layer normalization of AdaSpeech, which supplies our method. That paper pre-trained on LibriTTS and adapted to new voices by tuning about 5,000 per-speaker scale and bias parameters, reaching a speaker-similarity MOS of 3.96 versus 3.34 for tuning a speaker embedding alone.

On top of this backbone we add three pieces. The first is a per-utterance prosody summary: pooled pitch, energy, speaking rate and trailing pause, plus a sentence text embedding. The second is a shared dialogue GRU that reads these summaries turn by turn. The third is a per-participant GRU state, so each speaker's own trajectory can be followed. The GRU output conditions the variance predictors of the next utterance.

To keep identity out of the state, a speaker classifier is attached to the GRU state through gradient reversal, which pushes the state to be uninformative about who is speaking. We then measure leakage directly with a held-out linear probe rather than asserting it.

Pre-training proceeds in two stages. Stage one uses LibriTTS, described in LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech as 585 hours from 2,456 speakers split at sentence boundaries. We train on runs of consecutive sentences from the same chapter. This teaches monologue-scale prosodic continuity, the regime Simple and Effective Multi-sentence TTS with Expressive and Coherent Prosody addressed with fixed chunks of about 24 seconds. We assume utterance ordering within chapters is recoverable from the corpus metadata, and we will verify this before committing. Stage two uses the multi-speaker dialogue data of NCSSD from Generative Expressive Conversational Speech Synthesis, 236 hours across 776 speakers, which teaches turn-taking dynamics between different voices. Adaptation is then run on DailyTalk.

The analysis side asks what the state encodes. We probe it for turn index, emotion and speech-act labels, which DailyTalk annotates. We also test whether its gate values rise at topic shifts, which would indicate learned forgetting rather than simple averaging.

Baselines

The naive reference is FastSpeech 2 with multi-speaker pre-training and speaker conditioning but no dialogue context, fine-tuned on DailyTalk. The DailyTalk paper reports the corresponding context-free pairwise MOS as 3.86, against 3.90 for its context-aware version. That small gap is itself a motivation: context barely helped when learned from 20 hours alone.

The strongest published competitor that is directly comparable is H4C-TTS. It uses a FastSpeech 2 backbone with Sentence-BERT and wav2vec 2.0 turn encoders, separate recent and historical attention, and mixed-context layer normalization. It reports MOS 4.09, F0 RMSE 0.2978 and MCD 5.4624 on DailyTalk. MFCIG-CSS reports N-DMOS 3.980, P-DMOS 3.899 and MCD 9.53 with graph encoders over turns. Its MCD differs sharply from H4C-TTS's, which shows these numbers are not comparable across papers. We will therefore reimplement H4C-TTS on our backbone, vocoder and data splits rather than quote it. GPT-Talker (N-DMOS 3.89) is a discrete-token language model with a different generation pipeline. We include it only as a reference point in listening tests if released checkpoints permit, and otherwise report no head-to-head result for it.

The key ablations give every context method the same LibriTTS and NCSSD pre-training, parameter count within 10%, and hyperparameter search budget. These context methods are H4C-style attention, attention over all past turn summaries, and our GRU state. A further ablation trains the GRU from scratch on DailyTalk only, which isolates the value of multi-speaker pre-training. A last ablation removes gradient reversal, which isolates the decoupling. Beating H4C-TTS means a statistically significant CMOS preference plus lower F0 RMSE on turns six and later, with no loss in speaker similarity and no loss on early turns.

Why not just…?

Why not just give an attention model the whole dialogue history? This is the real threat. A dialogue has tens of turns, not thousands, so attention over one summary vector per turn is cheap, and the memory argument for recurrence largely disappears. GPT-Talker's degradation beyond three turns concerns serializing full speech-token sequences, not attention over compact summaries. We therefore run exactly this model with identical pre-training. If it matches the GRU, we will report that the transferable ingredient is the pre-trained cross-utterance prosody objective, not recurrence. That outcome would still answer the question left open by the finding that pre-training transfers only text prosody.

Why not just use the H4C-TTS design with more data? H4C-TTS hard-codes a recent/historical split and was trained only within DailyTalk. Pre-training it on NCSSD is part of our baseline set, so any gain we report is net of scale. The residual question is whether a learned forget gate beats a fixed two-way split. We test that on dialogues re-segmented so that the relevant context lies at varying distances.

Why not just rely on a speaker embedding and let the decoder sort out context? AdaSpeech's 3.34 versus 3.96 speaker-similarity gap already shows that a lone embedding adapts identity poorly. Moreover, without an explicit identity-free state, context features can carry one speaker's timbre into the other speaker's turns. Our leakage probe and turn-by-turn similarity measurements quantify exactly this failure in the no-reversal ablation.

Decisive Checks

All systems share one vocoder and are evaluated on DailyTalk test dialogues with metrics stratified by turn index. The metrics are CMOS against H4C-TTS, F0 RMSE, MCD, and inter-utterance pause error in the style of Makarov et al. We add speaker-similarity drift, measured as the cosine between a speaker-verification embedding of each turn and that speaker's first turn. DailyTalk localized context gains after turn five, so the headline test compares turns six onward.

Generalization to unseen voices is checked on held-out NCSSD speakers, using per-speaker parameters adapted from 20 sentences, AdaSpeech's regime.

The proposal is supported under three conditions. First, the pre-trained GRU beats the DailyTalk-only GRU on late turns. Second, it beats attention with matched pre-training by a significant CMOS margin. Third, identity cannot be decoded from its state above chance while similarity drift stays flat. It is refuted if pre-training gives no late-turn gain, which would mean discourse prosody does not transfer across speakers. It is also refuted if matched attention equals it, in which case the recurrent part is unnecessary.