Conversational speech synthesis tries to make a voice sound right for the moment: calmer after a sad remark, brighter when answering a joke, matching the pace of the other speaker. The best current systems get this context by building a graph over the dialogue history, with nodes for each past turn's text, its recorded audio, its speaker, and its emotion. They then read the tone of the conversation off the recorded audio of those past turns. That design breaks in exactly the settings where conversational synthesis is most useful. When rendering a dialogue script or an audiobook, no earlier turn has audio yet. In a live agent, the user's audio is often noisy or reverberant, or missing altogether. We propose to train a text-only "imagination" model to infer the acoustic style each past turn would have had, from the words and the conversational structure around it. The model is trained by alternating distillation against an audio-using graph teacher, so that at test time the dialogue graph can run with no reference audio at all.
The central bet is that a past turn's prosody (its pitch, energy, and rhythm) is predictable much better from its position in the conversation than from its own text alone. Speakers carry their own tone forward from turn to turn and partly mirror each other. Inferring all the missing audio nodes jointly over the graph should therefore recover far more of the "oracle audio" benefit than inferring each turn in isolation. This claim is directly testable, and if it fails, the graph machinery is unnecessary.
Where this sits
Graph-based conversational synthesis is the motivating line of work. Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling (ECSS) builds a five-node-type graph over text, audio, speaker, emotion category and intensity, and encodes it with a Heterogeneous Graph Transformer, a graph network that learns separate attention weights for each node and edge type. Multimodal Fine-grained Context Interaction Graph Modeling for Conversational Speech Synthesis (MFCIG-CSS) adds word-level semantic and prosody interaction graphs over TOD-BERT and Wav2Vec 2.0 features. Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis retrieves stored dialogues by semantic and acoustic-style similarity. Spoken Style Learning with Multi-modal Hierarchical Context Encoding predicts style-token weights from a DialogueGCN over turns. All four take context-turn audio as an input at inference. We keep their graph and synthesizer designs and remove that dependency.
Two observations motivate removing it. Donghyun Seong et al. show that reference audio leaks noise, reverberation and timbre into the output. Kangxiang Xia et al. show that static references impose rigid templates that suppress natural conversational inflection.
The training method comes from hybrid text-graph learning. GLEM alternates between training a language model and a graph network, each learning from the other's predictions on unlabeled nodes. After this mutual distillation, its language model alone predicts well on nodes whose edges are missing. We reuse that alternating structure but change what is missing and what is distilled. GLEM handles missing edges and distills class pseudo-labels. Here the missing parts are whole audio nodes, and the distilled targets are continuous acoustic-style embeddings. The file lists that continuous setting as unexplored for GLEM.
For the student's architecture we borrow from GraphFormers, which nests neighbor aggregation inside Transformer layers. Its one-directional neighbor attention lets earlier turns be encoded once and cached. GraphFormers also had to mask node text during pre-training to stop the language model from ignoring the graph, a failure we expect to face too. GL-Fusion offers deeper cross-attention fusion, but the file notes its memory cost on dense multi-turn graphs, so we treat it as an optional upper variant rather than the default.
Two findings constrain the design. ECSS reports that context utility peaks around 10 turns. SAT found that 3-hop neighborhoods hurt relative to 2-hop. Both caution against wide receptive fields, so the student's graph is limited to a 10-turn window with at most two hops of message passing.
Core Claim
Let $a_j$ be the acoustic-style embedding that an audio encoder extracts from the recording of past turn $j$, and let $\hat a_j$ be the student's estimate of it from text and graph structure only. Write $G$ for the dialogue graph of transcripts and speakers within the window, and $\mathcal{N}(j)$ for the neighbors of turn $j$ in that graph. The student is trained to minimize
$$\mathcal{L}_{\text{img}} = \sum_j \lVert \hat a_j(G) - a_j \rVert^2 + \lambda\, \mathrm{KL}\big(p_T(y \mid \hat a, G)\,\Vert\, p_T(y \mid a, G)\big),$$
where $p_T$ is the teacher's distribution over the current turn's emotion and intensity, $y$ is that label, and $\lambda$ weights the second term. The second term asks that imagined audio leads the teacher to the same decision as real audio. That matters more than matching every dimension of $a_j$, much of which encodes channel and timbre we do not want to reproduce.
The claim is that $\hat a_j(G)$, computed jointly over speaker and reply edges, closes a large fraction of the gap between a text-only graph and the oracle-audio graph. The comparison point is $\hat a_j(t_j)$, computed from turn $j$'s own text $t_j$ alone, which should close substantially less.
Research Program
We first train an ECSS-style teacher on DailyTalk using the ECSS emotion and intensity annotations, with real audio nodes.
The alternating loop then runs as follows. In the student step, the imagination model is a TOD-BERT encoder with nested speaker-edge and reply-edge aggregation and cached earlier turns. It is fit to $\mathcal{L}_{\text{img}}$, with a random subset of turn texts masked so that it must use neighbors. In the teacher step, the teacher is fine-tuned on graphs in which a random fraction of audio nodes is replaced by student imaginations. The teacher thereby learns to consume imagined nodes, and its revised outputs feed the next student step. As in GLEM, we will monitor for error amplification across rounds and cap the number of rounds by validation prosody error.
At inference, every audio node comes from the student, and the synthesizer is unchanged. A second mode handles noisy input. Real but corrupted user audio is gated against the imagination by a learned confidence weight, so clean real audio can still be used.
Analysis has three parts. We measure per-turn imputation quality as the correlation of $\hat a_j$ with $a_j$ on held-out dialogues. We break that quality down by edge type. We test whether imagined embeddings carry less noise and reverberation information than encoder outputs on corrupted audio, using a linear probe for room condition.
Baselines
The do-nothing reference point is the same teacher architecture with audio nodes deleted and retrained on text, speaker and emotion nodes only. This is the honest reference-free version of ECSS. Beating it is necessary but not sufficient.
The upper reference is the oracle teacher with clean recorded audio for all context turns. We report results as the fraction of the gap closed: (student − text-only) / (oracle − text-only), computed on the objective metrics.
The strongest existing system is MFCIG-CSS. On DailyTalk it reports N-DMOS 3.980, P-DMOS 3.899, pitch error MAE-P 0.439 and MCD 9.53 (MFCIG-CSS). ECSS reports N-DMOS 3.506 and E-DMOS 3.619 (ECSS). These DMOS values come from different listening tests and are not comparable to each other or to ours. We will therefore retrain MFCIG-CSS and ECSS on our DailyTalk split with the same FastSpeech 2 synthesizer, vocoder and training steps, and the same hyperparameter search budget. Each runs both with oracle audio and with audio removed.
Beating MFCIG-CSS means two things: lower MAE-P and MCD than its audio-removed variant, and staying within its oracle-audio pitch error plus a small margin. On noisy context audio, it means beating its oracle-mode numbers outright.
The ablation that isolates our key ingredient is per-turn imputation. It uses the same student encoder and the same loss, but graph aggregation is disabled, so each $\hat a_j$ depends only on $t_j$. Comparing it with the full student tests the central bet directly.
Why not just…?
Why not just impute each turn's audio embedding from its own text, with no graph? This is the most serious threat, and it is the per-turn ablation above. If its gap-closed fraction is within noise of the joint student, the graph contributes nothing to imputation, and the paper's result becomes "a text-to-style regressor is enough." We would report it that way. Two things argue the graph should matter. ECSS and the DialogueGCN-based system both credit speaker-inertia and inter-speaker edges for their gains. Short turns such as "okay" or "really?" carry almost no prosodic information in their text. We will report the comparison separately on turns of five words or fewer, where the difference should be largest.
Why not just synthesize the dialogue turn by turn and feed each generated turn back as the next turn's reference? For script rendering this rollout is natural, and the Controllable Context-Aware Conversational Speech Synthesis setup implicitly assumes previous-turn audio of this kind. Its weakness is that errors compound: a flat early turn makes later turns flatter. DailyTalk reports that sequential models degrade past the fifth turn. We run rollout as a full baseline with the oracle teacher and plot pitch error against turn index. If rollout matches our method at late turns, imagination offers only a speed advantage, and we would state that.
Why not just flatten the history into a prompt for an LLM-based TTS? The file documents losses from linearizing structure, both in graph reasoning (SAT) and in multi-party dialogue (Zhang et al.). However, no reported result on DailyTalk prosody for this approach was found on Lacuna. We include a controlled variant to test it: the student's encoder without graph layers, reading the serialized 10-turn history.
Proposed Experiments
The main experiment uses DailyTalk with ECSS annotations. We report MCD, MAE-P, energy and duration error, the emotion and intensity accuracy of an external classifier on generated speech, and N-DMOS, P-DMOS and E-DMOS from a single shared listening test. All conditions are evaluated with real context audio present at 100%, 50% and 0%.
A contamination experiment adds reverberation and babble noise to context audio at several signal-to-noise ratios. It compares oracle-on-noisy, imagination-only and gated systems. The prediction is that imagination plus gating beats oracle-on-noisy once noise is moderate.
A generalization test on ECC moves to multi-speaker educational conversations. DailyTalk has only one male and one female speaker, so this is where speaker edges are richer.
A context-swap check pairs each target turn with a mismatched dialogue history. If imagined context actually drives prosody, as it should, prosody error should rise as much as it does for the oracle teacher.
The decisive results are these. The joint student must close clearly more of the oracle gap than per-turn imputation, especially on short turns. It must beat turn-by-turn rollout at later turn positions. It must exceed oracle-on-noisy under realistic corruption. If the first fails, the graph-imputation thesis is wrong, regardless of how the other two come out.
References
- Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling
- Multimodal Fine-grained Context Interaction Graph Modeling for Conversational Speech Synthesis
- Retrieval-Augmented Dialogue Knowledge Aggregation for expressive conversational speech synthesis
- Spoken Style Learning with Multi-modal Hierarchical Context Encoding for Conversational Text-to-Speech Synthesis
- TSP-TTS: Text-based Style Predictor with Residual Vector Quantization for Expressive Text-to-Speech
- The ISCSLP 2024 Conversational Voice Clone (CoVoC) Challenge: Tasks, Results and Findings
- Learning on Large-scale Text-attributed Graphs via Variational Inference
- GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph
- GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model
- Enhancing Large Language Model for Knowledge Graph Completion via Structure-Aware Alignment-Tuning
- Controllable Context-Aware Conversational Speech Synthesis
- DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech
- Modeling both Context- and Speaker-Sensitive Dependence for Emotion Detection in Multi-speaker Conversations