Abstract
In recent years, large language models (LLMs) have achieved remarkable advancements in multimodal processing. However, existing task-oriented dialogue (TOD) datasets are predominantly text-based or use "clean" speech, lacking the real-world complexities like disfluencies and speaker variations. We introduce RealTalk-CN, the first Chinese multi-turn, multi-domain speech-text dual-modal TOD dataset. Comprising 5.4k dialogues (150 hours) with paired speech-text annotations, it captures diverse scenarios with spontaneous speech disfluencies. We also propose a novel cross-modal chat task simulating dynamic switching between speech and text. Our evaluations reveal significant robustness gaps in current state-of-the-art models when faced with realistic speech.
Introduction: The Gap in Task-Oriented Dialogue
While "omni-modal" models like GPT-4o and Qwen2-Audio have emerged, benchmarks for evaluating them in realistic settings have lagged. Most TOD datasets—where users aim to complete specific tasks like booking travel—rely on text or synthetic, clean speech. Real human interaction is messy, filled with hesitations, repetitions, and regional accents. Furthermore, users often switch modalities mid-conversation (e.g., voice queries followed by text inputs). RealTalk-CN addresses these gaps by providing a benchmark that tests for robustness against stuttering, accents, and modality shifts.
The RealTalk-CN Dataset
RealTalk-CN is a large-scale resource featuring 5.4k dialogues and 60k utterances across 58 domains, 55 intents, and 115 slot types. To ensure the benchmark reflects real-world usage, the authors recruited 300 volunteers with diverse backgrounds.
The dataset stands out by including a wide age range (18-54) and various regional Chinese accents, which are critical for testing demographic bias and model fairness. This diversity allows for the identification of specific performance gaps, such as the "Aging Gap," where models struggle with speech from older demographics.
Modeling Spontaneous Speech
A core contribution of RealTalk-CN is its rigorous annotation of spontaneous disfluencies. Unlike standard datasets that provide clean transcripts, this benchmark specifically labels four types of speech irregularities that frequently occur in natural conversation.
These annotations allow researchers to pinpoint exactly where models fail—whether it is a prolonged sound (Modal Particle Drag) or a user correcting themselves mid-sentence. This level of detail is essential for improving the robustness of speech-based LLMs in task-oriented scenarios.
Key Insights and Results
Extensive experiments comparing traditional "Pipeline" approaches (ASR + text LLM) against "End-to-End" multimodal models yielded several critical findings:
- The "Semantic vs. Detail" Divide: Disfluencies have a minimal impact on a model's ability to understand general intent but significantly degrade Slot Filling performance. Extracting specific details like phone numbers or names becomes much harder when speech is non-fluent.
- Pipeline vs. End-to-End: Traditional pipelines (e.g., Whisper + GPT-4o) lead in high-level reasoning and intent classification (53-62% accuracy). However, end-to-end models like Baichuan-Audio often prove more adaptable to raw audio quality, outperforming pipelines in extracting fine-grained details in multi-domain scenarios.
- Modality Switching as a Stress Test: In the novel cross-modal chat task, some models (like Baichuan-Omni) performed better when speech turns were interspersed with text, as text acts as a "clean" anchor. However, others struggled with "contextual amnesia" when switching modalities. GPT-4o-Audio-mini led this category with a score of 8.73/10.
- The Aging Gap: A significant performance drop (over 20%) was observed when processing speech from older demographics (ages 45-54), highlighting a lack of inclusivity in current training data.
Conclusion
RealTalk-CN exposes the robustness gaps in current speech-based LLMs. By providing a dataset that mirrors the complexity of real-world Chinese dialogue—including disfluencies, demographic diversity, and cross-modal interactions—it establishes a new standard for evaluating the next generation of conversational AI. Future work will focus on expanding dialect coverage and improving model fairness across all age groups and regional accents.
Summary updated