Retrieval-Augmented Dialogue Knowledge Aggregation for expressive conversational speech synthesis

Rui Liu, Zhenqi Jia, Feilong Bao, Haizhou Li

TL;DR

Researchers have introduced RADKA-CSS, a novel framework that brings Retrieval-Augmented Generation (RAG) to the world of Conversational Speech Synthesis (CSS). By retrieving and aggregating "style knowledge" from a database of similar past dialogues, the model can synthesize more expressive, empathetic, and contextually appropriate speech than models relying solely on the current conversation's history.

What Problem Does This Paper Address?

Conversational Speech Synthesis (CSS) aims to generate speech that sounds natural within a dialogue. While standard Text-to-Speech (TTS) focuses on single sentences, CSS must account for the "vibe" or prosody of an ongoing conversation.

The current bottleneck in CSS is that most models only look at the Current Dialogue (CD) history. However, human speakers often draw on a lifetime of social "knowledge"—knowing how a hotel reservation or a sympathetic apology should sound based on similar past experiences. Previous models lacked this external reference, often resulting in speech that, while clear, lacks the nuanced emotional alignment needed for truly empathetic AI agents.

Methods

The authors propose the Retrieval-Augmented Dialogue Knowledge Aggregation (RADKA-CSS) framework, which consists of four main stages:

  1. Database Preparation (SDSSD): They built a "Stored Dialogue Semantic-Style Database." This isn't just text; it stores both the semantic meaning (via text summaries) and the acoustic style (via audio vectors) of thousands of dialogue fragments.
  2. Multi-attribute Retrieval: When the model needs to generate a response, it doesn't just look at the text. It uses a dual-retrieval scheme to find past dialogues that match the current one in both scenario (what is being said) and style (how it's being said).
  3. Multi-granularity Heterogeneous Graph (MgHG): To process these dialogues, the authors designed a sophisticated graph structure. It models relationships at three levels: word-level (local impact), sentence-level (conversational flow), and dialogue-level (overall theme). This allows the model to capture both the "forest" and the "trees" of conversational style.
  4. Knowledge Aggregation & Contrastive Learning: The model combines style info from the retrieved examples, the current history, and a predicted style vector. They also used Contrastive Learning during training to ensure the model's internal "map" of dialogues correctly clusters similar styles together while pushing different ones apart.

Datasets

The study primarily uses the DailyTalk dataset, a benchmark for CSS.

  • DailyTalk: Contains ~20 hours of high-quality recordings (23,773 clips) of a male and female speaker in over 2,500 conversations.
  • The authors highlight that DailyTalk is ideal for RAG because its consistent two-speaker format allows for clean database construction and retrieval without the noise of massive multi-speaker datasets.

Key Insights and Ideas

  • Style is as important as Semantics: The paper demonstrates that retrieving based on text alone isn't enough. By matching the audio style of the current dialogue to stored examples, the model achieves much better prosodic alignment.
  • RAG isn't just for LLMs: While RAG is famous for helping Large Language Models avoid "hallucinations" by looking up facts, this paper proves it is equally effective for "looking up" non-verbal social cues and prosody in speech synthesis.
  • The Power of Graphs: Moving from simple linear encoders to heterogeneous graphs (MgHG) was a major win. Capturing "parent-child" (words in a sentence) and "sibling" (adjacent sentences) relationships allows the model to understand the structural and temporal nuances of human speech.

Key Results

RADKA-CSS outperformed five state-of-the-art baselines (including ECSS and CONCSS) across all metrics:

  • Subjective Naturalness (N-DMOS): Achieved a score of 3.904, significantly higher than the best baseline (ECSS at 3.720).
  • Style Alignment (S-DMOS): Achieved 3.879, proving the synthesized speech matched the conversational context better than previous methods.
  • Objective Accuracy: The model showed the lowest Mean Absolute Error (MAE) for pitch (MAE-P), energy (MAE-E), and duration (MAE-D), indicating its predicted speech patterns were closest to real human ground truth.
  • Ablation Success: Removing the "Stored Dialogue" (the RAG component) caused the largest performance drop, confirming that external knowledge is the "secret sauce" of this model.

Limitations and Future Work

The authors acknowledge a few key areas for improvement:

  • Fixed-User Focus: The current model was tested on a two-speaker dataset (DailyTalk). It performs well in stable user-agent interactions but isn't yet optimized for "public-facing" scenarios where many different speakers might alternate rapidly.
  • Computational Complexity: Using a multi-granularity graph and a retrieval module adds overhead. Future work may need to focus on optimizing these for real-time, low-latency applications.
  • Expansion: The team aims to extend RADKA-CSS to handle more complex, multi-party conversational environments in the future.

Summary updated

Abstract

Highlights•We propose a novel Retrieval-Augmented Dialogue Knowledge Aggregation CSS model, termed RADKA-CSS.•We introduce a dialogue semantic-style database and design a multi-attribute retrieval module to facilitate style- and semantics-based dialogue retrieval.•We propose a multi-granularity heterogeneous graph modelling mechanism that aims to capture the structural and temporal relations of nodes at different granularities, effectively encoding dialogue semantics and style information.•The experimental results on the DailyTalk benchmarking dataset validate the effectiveness of RADKA-CSS.
BibTeX
@article{liu2025retrieval,
  author = {Rui Liu and Zhenqi Jia and Feilong Bao and Haizhou Li},
  title = {{Retrieval-Augmented Dialogue Knowledge Aggregation for expressive conversational speech synthesis}},
  journal = {Information Fusion},
  year = {2025},
  doi = {10.1016/j.inffus.2025.102948},
}

Loading related content...

Figure