RealTalk-CN: A Realistic Chinese Speech-Text Dialogue Benchmark With Cross-Modal Interaction Analysis

Enzhi Wang, Qicheng Li, Shiwan Zhao, Aobo Kong, Jiaming Zhou, Xi Yang, Yequan Wang, Yonghua Lin, Yong QinarXiv 2025 · arXiv · OpenReview

Abstract

In recent years, large language models (LLMs) have achieved remarkable advancements in multimodal processing. However, existing task-oriented dialogue (TOD) datasets are predominantly text-based or use "clean" speech, lacking the real-world complexities like disfluencies and speaker variations. We introduce RealTalk-CN, the first Chinese multi-turn, multi-domain speech-text dual-modal TOD dataset. Comprising 5.4k dialogues (150 hours) with paired speech-text annotations, it captures diverse scenarios with spontaneous speech disfluencies. We also propose a novel cross-modal chat task simulating dynamic switching between speech and text. Our evaluations reveal significant robustness gaps in current state-of-the-art models when faced with realistic speech.

Introduction: The Gap in Task-Oriented Dialogue

While "omni-modal" models like GPT-4o and Qwen2-Audio have emerged, benchmarks for evaluating them in realistic settings have lagged. Most TOD datasets—where users aim to complete specific tasks like booking travel—rely on text or synthetic, clean speech. Real human interaction is messy, filled with hesitations, repetitions, and regional accents. Furthermore, users often switch modalities mid-conversation (e.g., voice queries followed by text inputs). RealTalk-CN addresses these gaps by providing a benchmark that tests for robustness against stuttering, accents, and modality shifts.

The RealTalk-CN Dataset

RealTalk-CN is a large-scale resource featuring 5.4k dialogues and 60k utterances across 58 domains, 55 intents, and 115 slot types. To ensure the benchmark reflects real-world usage, the authors recruited 300 volunteers with diverse backgrounds.

Demographic distribution of speakers in the RealTalk-CN dataset, covering age, gender, and geography.

The dataset stands out by including a wide age range (18-54) and various regional Chinese accents, which are critical for testing demographic bias and model fairness. This diversity allows for the identification of specific performance gaps, such as the "Aging Gap," where models struggle with speech from older demographics.

Modeling Spontaneous Speech

A core contribution of RealTalk-CN is its rigorous annotation of spontaneous disfluencies. Unlike standard datasets that provide clean transcripts, this benchmark specifically labels four types of speech irregularities that frequently occur in natural conversation.

Examples of the four types of speech disfluencies annotated in RealTalk-CN: Modal Particle Drag, Repetition, Self-Correction, and Hesitation.

These annotations allow researchers to pinpoint exactly where models fail—whether it is a prolonged sound (Modal Particle Drag) or a user correcting themselves mid-sentence. This level of detail is essential for improving the robustness of speech-based LLMs in task-oriented scenarios.

Key Insights and Results

Extensive experiments comparing traditional "Pipeline" approaches (ASR + text LLM) against "End-to-End" multimodal models yielded several critical findings:

  • The "Semantic vs. Detail" Divide: Disfluencies have a minimal impact on a model's ability to understand general intent but significantly degrade Slot Filling performance. Extracting specific details like phone numbers or names becomes much harder when speech is non-fluent.
  • Pipeline vs. End-to-End: Traditional pipelines (e.g., Whisper + GPT-4o) lead in high-level reasoning and intent classification (53-62% accuracy). However, end-to-end models like Baichuan-Audio often prove more adaptable to raw audio quality, outperforming pipelines in extracting fine-grained details in multi-domain scenarios.
  • Modality Switching as a Stress Test: In the novel cross-modal chat task, some models (like Baichuan-Omni) performed better when speech turns were interspersed with text, as text acts as a "clean" anchor. However, others struggled with "contextual amnesia" when switching modalities. GPT-4o-Audio-mini led this category with a score of 8.73/10.
  • The Aging Gap: A significant performance drop (over 20%) was observed when processing speech from older demographics (ages 45-54), highlighting a lack of inclusivity in current training data.

Conclusion

RealTalk-CN exposes the robustness gaps in current speech-based LLMs. By providing a dataset that mirrors the complexity of real-world Chinese dialogue—including disfluencies, demographic diversity, and cross-modal interactions—it establishes a new standard for evaluating the next generation of conversational AI. Future work will focus on expanding dialect coverage and improving model fairness across all age groups and regional accents.

Summary updated

Abstract

In recent years, large language models (LLMs) have achieved remarkable advancements in multimodal processing, including end-to-end speech-based language models that enable natural interactions and perform specific tasks in task-oriented dialogue (TOD) systems. However, existing TOD datasets are predominantly text-based, lacking real speech signals that are essential for evaluating the robustness of speech-based LLMs. Moreover, existing speech TOD datasets are primarily English and lack critical aspects such as speech disfluencies and speaker variations. To address these gaps, we introduce RealTalk-CN, the first Chinese multi-turn, multi-domain speech-text dual-modal TOD dataset, comprising 5.4k dialogues (60K utterances, 150 hours) with paired speech-text annotations. RealTalk-CN captures diverse dialogue scenarios with annotated spontaneous speech disfluencies, ensuring comprehensive coverage of real-world complexities in speech dialogue. In addition, we propose a novel cross-modal chat task that authentically simulates real-world user interactions, allowing dynamic switching between speech and text modalities. Our evaluation covers robustness to speech disfluencies, sensitivity to speaker characteristics, and cross-domain performance. Extensive experiments validate the effectiveness of RealTalk-CN, establishing a strong foundation for Chinese speech-based LLMs research.
BibTeX
@misc{wang2025realtalk,
  author = {Enzhi Wang and Qicheng Li and Shiwan Zhao and Aobo Kong and Jiaming Zhou and Xi Yang and Yequan Wang and Yonghua Lin and Yong Qin},
  title = {{RealTalk-CN: A Realistic Chinese Speech-Text Dialogue Benchmark With Cross-Modal Interaction Analysis}},
  year = {2025},
  eprint = {2508.10015},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.10015},
}
Figure