DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech

Keon Lee, Kyumin Park, Daeyoung Kim

TL;DR

Researchers from KAIST and KRAFTON have released DailyTalk, the first high-quality, open-source dataset specifically designed for conversational Text-to-Speech (TTS). By recording over 2,500 dialogues with professional actors and a context-aware baseline model, they demonstrate that incorporating dialogue history significantly improves the naturalness of synthesized speech in multi-turn conversations.

What Problem Does This Paper Address?

Standard TTS systems are typically trained on datasets consisting of isolated, individual utterances (like LJSpeech). While these models excel at reading single sentences clearly, they often fail in a conversational setting. In real life, how we say something depends heavily on what was said before—the "prosodic flow," emotional context, and even the use of filler words like "uh" or "um."

Existing conversational datasets often suffer from two major issues: they are either private (held by big tech companies) or they are "in-the-wild" recordings (like podcasts) that contain inconsistent audio quality and background noise. The authors created DailyTalk to provide a clean, studio-quality, and publicly available benchmark to bridge this gap.

Methods

The authors developed both a dataset and a baseline architecture:

  1. The Dataset Pipeline: They sampled 2,541 dialogues from the text-based DailyDialog dataset. They refined these scripts to ensure they were "dyadic" (between one male and one female speaker) and had at least five turns to ensure sufficient context.
  2. The Baseline Model: They built upon FastSpeech2, a popular non-autoregressive TTS model. To make it "context-aware," they added a Context Encoder that processes previous dialogue history.
  3. Alignment: They replaced the traditional Montreal Forced Aligner with a CTC-based aligner, allowing the model to learn durations in an end-to-end manner, which is more robust for expressive, conversational speech.

Datasets

  • DailyTalk (New): 20 hours of studio-recorded speech across 2,541 dialogues. It includes annotations for emotion, speech acts, and topics. Crucially, it includes "filling-gap" words (fgs) like uh and umm in about half the dialogues to mimic natural speech.
  • LJSpeech & VCTK: Used as "integrity" benchmarks to prove that DailyTalk is high-quality enough to train standard TTS models.
  • DailyDialog: The original text-only dataset from which the scripts were derived.

Key Insights and Ideas

  • Context "Piles Up": The researchers found that the benefit of context-aware modeling becomes more apparent as a conversation progresses. They split their evaluation at "Turn 5," finding that the baseline model significantly outperformed non-contextual models in the later stages of a dialogue.
  • Fillers Matter: By explicitly directing actors to include natural fillers (uh, um), the dataset allows models to learn the rhythmic "hesitations" that make human conversation feel authentic.
  • New Evaluation Metrics: Since standard MOS (Mean Opinion Score) only looks at single sentences, the authors proposed four new metrics to evaluate dialogue, such as User-System Simulation MOS, where one speaker is ground truth and the other is synthesized.

Key Results

  • Integrity: DailyTalk achieved a Mean Opinion Score (MOS) of 3.82, comparable to the gold-standard LJSpeech (3.85), proving it is a high-quality dataset for general TTS.
  • Conversational Superiority: In "Pairwise MOS" tests (where listeners compare two adjacent utterances), the context-aware baseline scored 3.90, outperforming the standard FastSpeech2 (3.86).
  • Turn-Based Performance: In the "after turn 5" analysis, the context-aware model showed a "huge gain" in naturalness compared to the standard model, which actually saw a slight degradation in performance as the dialogue grew longer and more complex.

Limitations and Future Work

  • Speaker Diversity: The current version of DailyTalk features only two speakers (one male, one female). While this is a great start for dyadic research, it doesn't capture the variety of voices found in larger datasets.
  • Language & Culture: The dataset is currently limited to English-fluent speakers. The authors suggest that future work should expand into more spontaneous features and diverse cultural linguistic patterns.
  • Complexity vs. Quality: The authors noted that their baseline model, while more contextually aware, had slightly higher complexity which can sometimes lead to minor trade-offs in raw audio quality compared to simpler models.

Summary updated

Abstract

The majority of current Text-to-Speech (TTS) datasets, which are collections of individual utterances, contain few conversational aspects. In this paper, we introduce DailyTalk, a high-quality conversational speech dataset designed for conversational TTS. We sampled, modified, and recorded 2,541 dialogues from the open-domain dialogue dataset DailyDialog inheriting its annotated attributes. On top of our dataset, we extend prior work as our baseline, where a non-autoregressive TTS is conditioned on historical information in a dialogue. From the baseline experiment with both general and our novel metrics, we show that DailyTalk can be used as a general TTS dataset, and more than that, our baseline can represent contextual information from DailyTalk. The DailyTalk dataset and baseline code are freely available for academic use with CC-BY-SA 4.0 license 1 .
BibTeX
@inproceedings{lee2023dailytalk,
  author = {Keon Lee and Kyumin Park and Daeyoung Kim},
  title = {{DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=Hezc4glO6a},
}

Loading related content...

Figure