Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis

Rui Liu, Zhenqi Jia, Feilong Bao, Haizhou Li

Abstract

Conversational speech synthesis (CSS) aims to take the current dialogue (CD) history as a reference to synthesize expressive speech that aligns with the conversational style. Unlike CD, stored dialogue (SD) contains preserved dialogue fragments from earlier stages of user-agent interaction, which include style expression knowledge relevant to scenarios similar to those in CD. Note that this knowledge plays a significant role in enabling the agent to synthesize expressive conversational speech that generates empathetic feedback. However, prior research has overlooked this aspect. To address this issue, we propose a novel Retrieval-Augmented Dialogue Knowledge Aggregation scheme for expressive CSS, termed RADKA-CSS, which includes three main components: 1) To effectively retrieve dialogues from SD that are similar to CD in terms of both semantic and style. First, we build a stored dialogue semantic-style database (SDSSD) which includes the text and audio samples. Then, we design a multi-attribute retrieval scheme to match the dialogue semantic and style vectors of the CD with the stored dialogue semantic and style vectors in the SDSSD, retrieving the most similar dialogues. 2) To effectively utilize the style knowledge from CD and SD, we propose adopting the multi-granularity graph structure to encode the dialogue and introducing a multi-source style knowledge aggregation mechanism. 3) Finally, the aggregated style knowledge are fed into the speech synthesizer to help the agent synthesize expressive speech that aligns with the conversational style. We conducted a comprehensive and in-depth experiment based on the DailyTalk dataset, which is a benchmarking dataset for the CSS task. Both objective and subjective evaluations demonstrate that RADKA-CSS outperforms baseline models in expressiveness rendering.

Introduction

Conversational Speech Synthesis (CSS) is a specialized branch of Text-to-Speech (TTS) that focuses on generating speech that sounds natural within the context of a dialogue. While standard TTS models often treat sentences in isolation, CSS must account for the ongoing "vibe," prosody, and emotional alignment of a conversation.

The primary challenge in current CSS research is the limited scope of context. Most existing models rely solely on the Current Dialogue (CD) history. However, human communication is deeply rooted in prior experience; we know how a hotel reservation or a sympathetic apology should sound because we have heard similar interactions before. To bridge this gap, researchers have introduced RADKA-CSS, a framework that incorporates Retrieval-Augmented Generation (RAG) into the CSS pipeline. By retrieving and aggregating "style knowledge" from a database of similar past dialogues, the model can synthesize more expressive and empathetic speech.

Comparison between traditional CSS and the proposed RADKA-CSS architecture.

Methodology: The RADKA-CSS Framework

The RADKA-CSS framework operates across four distinct stages to integrate external dialogue knowledge into the synthesis process.

1. Database Preparation (SDSSD)

The foundation of the system is the Stored Dialogue Semantic-Style Database (SDSSD). This database stores thousands of dialogue fragments, capturing both their semantic meaning (via text summaries generated by models like bart-large-cnn-samsum) and their acoustic style (represented as style vectors extracted from audio).

2. Multi-attribute Retrieval

When synthesizing a response, the system performs a dual-retrieval process. It queries the SDSSD to find past dialogues that match the current conversation in both scenario (what is being said) and style (how it is being said). This ensures the retrieved "knowledge" is contextually and prosodically relevant.

3. Multi-source Style Knowledge Aggregator

To process the complex relationships within dialogues, the authors utilize a Multi-granularity Heterogeneous Graph (MgHG). This structure models the conversation at three levels:

  • Word-level: Captures local linguistic impact.
  • Sentence-level: Models the conversational flow.
  • Dialogue-level: Captures the overall theme and emotional arc.

This hierarchical approach allows the model to understand both the structural and temporal nuances of human speech.

4. Speech Synthesizer and Contrastive Learning

The aggregated style knowledge—derived from the retrieved examples, the current history, and a predicted style vector—is fed into a speech synthesizer. During training, Contrastive Learning is employed to ensure the model's latent space correctly clusters similar styles while distinguishing between different ones.

The four-stage pipeline of RADKA-CSS, from database preparation to speech synthesis.

Experimental Evaluation

The researchers evaluated RADKA-CSS using the DailyTalk dataset, a benchmark for CSS containing approximately 20 hours of high-quality recordings between a male and female speaker across over 2,500 conversations.

Key Results

RADKA-CSS consistently outperformed five state-of-the-art baselines (including ECSS and CONCSS) across both subjective and objective metrics:

  • Subjective Naturalness (N-DMOS): RADKA-CSS achieved a score of 3.904, significantly higher than the top baseline (ECSS at 3.720).
  • Style Alignment (S-DMOS): The model scored 3.879, indicating superior prosodic alignment with the conversational context.
  • Objective Metrics: The model demonstrated the lowest Mean Absolute Error (MAE) for pitch (MAE-P), energy (MAE-E), and duration (MAE-D), confirming its high fidelity to human speech patterns.

Ablation studies further highlighted that the "Stored Dialogue" (RAG) component was the most critical factor in performance gains, proving that external knowledge is essential for expressive synthesis.

Conclusion and Future Work

RADKA-CSS demonstrates that RAG is not just for Large Language Models; it is a powerful tool for capturing non-verbal social cues and prosody in speech synthesis. By "looking up" how similar conversations have sounded in the past, AI agents can generate more empathetic and contextually appropriate feedback.

Future work will focus on optimizing the computational complexity of the graph-based retrieval module for real-time applications and expanding the framework to handle more complex, multi-party conversational environments beyond the two-speaker format of DailyTalk.

Summary updated

Abstract

Conversational speech synthesis (CSS) aims to take the current dialogue (CD) history as a reference to synthesize expressive speech that aligns with the conversational style. Unlike CD, stored dialogue (SD) contains preserved dialogue fragments from earlier stages of user-agent interaction, which include style expression knowledge relevant to scenarios similar to those in CD. Note that this knowledge plays a significant role in enabling the agent to synthesize expressive conversational speech that generates empathetic feedback. However, prior research has overlooked this aspect. To address this issue, we propose a novel Retrieval-Augmented Dialogue Knowledge Aggregation scheme for expressive CSS, termed RADKA-CSS, which includes three main components: 1) To effectively retrieve dialogues from SD that are similar to CD in terms of both semantic and style. First, we build a stored dialogue semantic-style database (SDSSD) which includes the text and audio samples. Then, we design a multi-attribute retrieval scheme to match the dialogue semantic and style vectors of the CD with the stored dialogue semantic and style vectors in the SDSSD, retrieving the most similar dialogues. 2) To effectively utilize the style knowledge from CD and SD, we propose adopting the multi-granularity graph structure to encode the dialogue and introducing a multi-source style knowledge aggregation mechanism. 3) Finally, the aggregated style knowledge are fed into the speech synthesizer to help the agent synthesize expressive speech that aligns with the conversational style. We conducted a comprehensive and in-depth experiment based on the DailyTalk dataset, which is a benchmarking dataset for the CSS task. Both objective and subjective evaluations demonstrate that RADKA-CSS outperforms baseline models in expressiveness rendering. Code and audio samples can be found at: https://github.com/Coder-jzq/RADKA-CSS.
BibTeX
@article{liu2025retrieval,
  author = {Rui Liu and Zhenqi Jia and Feilong Bao and Haizhou Li},
  title = {{Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis}},
  journal = {Information Fusion},
  year = {2025},
  doi = {10.1016/j.inffus.2025.102948},
}
Figure