TL;DR
This paper introduces InteractSpeech, a novel 150-hour English spoken dialogue dataset designed to address the critical lack of high-quality data for training nuanced spoken dialogue models. InteractSpeech features detailed annotations for complex interactional phenomena like interruptions and backchannels, enabling the development of more natural and responsive AI conversational agents.
What Problem Does This Paper Address?
Modern spoken dialogue models (SDMs) have made impressive strides, but they still struggle with the subtle, dynamic aspects of human conversation. These include handling interruptions gracefully, providing timely backchannels (like "uh-huh" or "I see"), and generally managing the flow of a multi-turn dialogue. A major roadblock to improving these capabilities is the scarcity of publicly available, meticulously annotated datasets that capture these fine-grained interactional dynamics. Existing datasets are often outdated, lack detailed timestamps, or focus on simpler command-based interactions, failing to provide the rich data needed to train AI that can truly participate in natural human-like conversations.
Methods
The core contribution is the creation of the InteractSpeech dataset. This was achieved through a multi-stage process:
- Synthetic Data Generation: They used a multi-stage prompt programming approach with GPT-40 to generate interactive dialogue scenarios. This involved first identifying suitable points in existing dialogue texts for interaction markers (like interruptions) and then generating natural-sounding content for these turns. This process was iteratively refined with dialogue experts.
- Speech Synthesis: Advanced text-to-speech (TTS) models were used to convert the generated interactive text into dual-track audio, complete with precise speaker timestamps. This allows for simulating various interaction timings, such as an interruption starting within a specific timeframe of the ongoing speech.
- Real-World Data Filtering: They also filtered existing real-world spoken dialogue corpora (specifically CANDOR) to extract interaction-dense segments. This involved sophisticated techniques like speaker diarization to detect overlaps, a vocal consistency check (timbre analysis) to distinguish between true turn-taking and backchannels, and temporal filtering to isolate high-quality interactive exchanges.
- Formal Framework: The dataset is underpinned by a formal framework for "Modeling Interaction Dynamics," which defines concepts like conversational turns, speech utterances, and silence periods, providing a principled basis for annotation.
Datasets
- InteractSpeech: The newly introduced dataset, comprising approximately 150 hours of spoken dialogue. It includes:
- A 148-hour in-domain training set.
- A 2-hour in-domain test set with manual speaker timestamps.
- A 1-hour out-of-domain (OOD) interactive test set where humans converse with existing dialogue systems.
- Source Data for Synthetic Generation: Dialog corpora like SODA, Dialogsum, PLACES, and MultiWOZ 2.2 were used as a basis for generating synthetic dialogues.
- Source Data for Real-World Filtering: The CANDOR corpus was used for filtering real-world interaction segments.
- Benchmarking Datasets: For comparison and evaluation, the paper references Fisher, Switchboard, Command FDM, and Voice FDM.
Key Insights and Ideas
- The critical need for interaction-specific data: The paper strongly argues that current SDMs are limited by the lack of datasets that explicitly capture nuanced interactional dynamics. Simply having more dialogue data isn't enough; it needs to be structured to teach models how to interact.
- Multi-stage prompt programming is key for synthetic data quality: Their experiments show that a multi-stage approach to prompting LLMs for dialogue generation significantly improves the naturalness, consistency, and specificity of the resulting interactive scenarios compared to a single-stage approach.
- Timbre consistency is a useful signal for distinguishing interaction types: Using timbre analysis to check if the speaker's voice changes after an overlap is an effective way to differentiate between a genuine interruption and a backchannel.
- InteractSpeech bridges the synthetic-real gap: The model fine-tuned on InteractSpeech shows strong generalization to real-world scenarios, suggesting the dataset effectively combines the benefits of synthetic data (scalability, control) with the realism of filtered real-world data.
Key Results
- Speech Understanding: A Qwen-Omni-FT model fine-tuned on InteractSpeech achieved 73.4% overall accuracy on the in-domain test set for classifying interactional events (Backchannel, Interruption, Gap, Pause). This significantly outperforms baselines like GPT-40 (54.2%) and zero-shot models.
- Generalization: The fine-tuned model also achieved 63.3% accuracy on the real-world scenarios (RWS) test set, demonstrating good out-of-domain generalization.
- Textual Scenario Evaluation: A LLaMA 3-8B model fine-tuned on the textual component of InteractSpeech achieved an 86% overall agreement rate with human evaluators for generating natural, consistent, and specific responses following scripted interruptions.
- Audio Quality: The InteractSpeech dataset's audio quality was rated highly (MOS of 4.35 ± 0.12), comparable to established corpora like Switchboard and Fisher, despite a significant portion being synthetically generated.
Limitations and Future Work
- Quantifying Holistic Improvement: The authors acknowledge that directly quantifying the improvement in holistic, end-to-end interaction modeling is challenging due to the need for substantial resources and proprietary architectures for full-duplex SDMs.
- Standardized Metrics: The field currently lacks standardized metrics to precisely evaluate intricate dialogue dynamics like turn-taking and interruption handling in fully integrated agents.
- Future Work: The paper suggests that future work, leveraging advancements in open-source models and evaluation methodologies, will be better positioned to demonstrate end-to-end improvements. They also highlight the need for continued development of robust, standardized metrics for interactional competence.
Summary updated