CMCC: A Comprehensive and Large-Scale Human-Human Dataset for Dialogue Systems

Yi Huang, Xiaoting Wu, Si Chen, Wei Hu, Qing Zhu, Junlan Feng, Chao Deng, Zhijian Ou, Jiangjiang Zhao2022 · DOI

Abstract

Most current Task-Oriented Dialogue (TOD) systems are trained on datasets like MultiWOZ, which are often small-scale and collected via crowd-workers following rigid scripts. These datasets fail to capture the messy reality of human interaction—specifically, the emotional friction that occurs when a real user has a problem. The CMCC dataset introduces a massive collection of 100,000 real-world human-human customer service dialogues, bridging the gap between task completion and emotional support. By annotating for user emotions and "care-oriented" service acts, the authors enable dialogue systems to move beyond dry task execution toward more empathetic, human-like interactions.

The CMCC Dataset

The primary contribution of this work is the China Mobile Customer Care (CMCC) dataset, which significantly expands upon previous human-human spoken dialogue sets.

  • Total Scale: Approximately 100,000 dialogues derived from real transcripts between users and service staff.
  • Labeled Subset: 8,975 dialogues manually annotated with a specialized care-oriented taxonomy.
  • Taxonomy: The researchers developed labels for 8 user emotions (e.g., "Emotionally More Agitated," "Dissatisfied with Business Rules") and 17 customer service acts, including four new "Expanded Caring Acts": Comfort, Empathy, Apology, and Suggestion.

Methodology: Markovian Generative Architecture

To leverage this data, the authors proposed a Markovian Generative Architecture (MGA) based on a Chinese GPT-2 backbone. Unlike standard models that map dialogue history directly to a response, the MGA "Variant Model" is designed to explicitly predict the user's emotion and the appropriate "caring" service act as intermediate steps before generating the final text response.

The MGA architecture incorporates user emotion and expanded caring acts into the generation process to ensure empathetic responses.

Analysis of Emotional Trajectories

A key insight from the CMCC dataset is the visualization of the "Healing Trajectory." By using Sankey diagrams to track emotion transitions across turns, the authors demonstrate how negative user emotions in the initial stages of a call can be successfully transitioned to "Neutral" or "Satisfied" states by the fourth turn, provided the agent employs appropriate caring service acts.

The Sankey diagram visualizes the "healing trajectory," showing how negative user emotions transition toward neutral states over multiple dialogue turns.

Analysis also revealed a strong correlation between specific emotions and task-oriented acts. For instance, the most frequent negative emotion, "Complain About A Problem," directly correlates with the "Inform" service act. This confirms that in real-world settings, emotional outbursts are usually tied to specific technical hurdles rather than being random.

Experimental Results

The experiments compared a baseline TOD model against the Variant Model utilizing the care-oriented annotations:

  • Task Success: The Variant Model achieved a Success Rate of 0.357, compared to 0.315 for the baseline—a significant improvement in real-world performance.
  • Fluency: The BLEU-4 score improved from 4.137 to 4.669, indicating more natural and contextually appropriate responses.
  • Emotion Recognition: Using a generation-based approach for emotion recognition improved the macro-F1 score from 0.301 to 0.393, proving much better at identifying rare but critical negative emotions in imbalanced data.

Conclusion

While the CMCC dataset is a massive step forward, the authors note that only about 9% of the 100,000 dialogues are currently labeled. Future work will focus on expanding these annotations and enriching the dataset with labels for specific solutions, external knowledge base links, and API calls to better model the technical "back-end" of task completion alongside emotional care.

Summary updated

BibTeX
@misc{huang2022cmcc,
  author = {Yi Huang and Xiaoting Wu and Si Chen and Wei Hu and Qing Zhu and Junlan Feng and Chao Deng and Zhijian Ou and Jiangjiang Zhao},
  title = {{CMCC: A Comprehensive and Large-Scale Human-Human Dataset for Dialogue Systems}},
  year = {2022},
  doi = {10.18653/v1/2022.seretod-1.7},
}
Figure