A Chit-Chats Enhanced Task-Oriented Dialogue Corpora for Fuse-Motive Conversation Systems

Changhong Yu, Chunhong Zhang, Qi Sun

TL;DR

This paper introduces CCET, a novel Chinese dialogue dataset that fuses task-oriented dialogue (TOD) and chit-chat (CC) utterances to create more robust conversational AI. They propose an automatic method for constructing such "fuse-motive" datasets, enabling TOD systems to handle the natural, mixed-motive interactions found in real-world conversations without costly manual annotation.

What Problem Does This Paper Address?

Current dialogue systems are often siloed: task-oriented dialogue (TOD) systems excel at achieving specific goals (like booking a flight), while open-domain chit-chat (CC) systems are good at engaging in general conversation. However, real-world user interactions are rarely so neatly categorized. Users often intersperse casual chat with task-related queries, leading to "undesirable failure" in existing TOD systems that are trained on clean, single-motive data. This disconnect between benchmark performance and real-world applicability is a significant challenge. Building systems that can seamlessly switch between or blend TOD and CC motives is crucial for practical deployment, but creating datasets that capture this "fuse-motive" behavior is difficult and expensive, often relying on manual annotation.

Methods

The authors propose an automatic, data-driven approach to create fuse-motive dialogue datasets. Their core idea is to leverage existing TOD and CC corpora and fuse them.

  1. Dialogue Structure Redefinition: They adapt the structure of the RisaWOZ dataset (a Chinese TOD benchmark) to accommodate fuse-motive interactions. This involves redefining turns into NLU (Natural Language Understanding), Query, and NLG (Natural Language Generation) components, with specific sub-fields like "Info Msg" and "Query Msg" to capture user intents and requests.
  2. Corpora Alignment: They align existing Chinese CC corpora (LCCC-base and NaturalConv) to this new structure. For CC turns, they mark "Info Msg" and "Query Msg" as empty, indicating no task-specific information is provided or requested, and set the "DB Status" to a special "<NotQuery>" category.
  3. Corpora Fusion: They then fuse the aligned TOD (RisaWOZ, denoted as 'T') and CC (LCCC-base/NaturalConv, denoted as 'C') corpora. This involves creating new samples by combining dialogue history from one corpus with current turn input-output pairs from the other (e.g., TC and CT). A filtering rule is applied for CT fusion to ensure contextual relevance, discarding fused turns where the original TOD context was too tightly coupled to be useful with CC input.

This process results in the CCET (Chinese Chat-Enhanced-Task) dataset. They also propose new evaluation metrics for fuse-motive systems, including adapted TOD metrics and a focus on dialog act F1 score for evaluating mixed-motive performance.

Datasets

  • RisaWOZ: A large-scale, multi-domain Chinese task-oriented dialogue dataset. This forms the basis for the TOD component of CCET.
  • LCCC-base: A large-scale Chinese conversation dataset for open-domain chatbots.
  • NaturalConv: A multi-turn Chinese CC dataset with longer sessions and topic-driven conversations.

These three datasets are used to construct the new CCET dataset, which is a fuse-motive dialogue corpus. The authors also mention that they truncate the larger CC corpora (LCCC-base and NaturalConv) to balance their size with the TOD corpus.

Key Insights and Ideas

  • The "Fuse-Motive" Gap: The paper clearly articulates the critical need for dialogue systems that can handle the natural blend of task-oriented and chit-chat interactions, a gap largely unaddressed by existing benchmarks and models.
  • Automatic Dataset Construction: The most significant contribution is the proposed automatic method for creating fuse-motive datasets. This bypasses the high cost and limitations of manual annotation, making it more feasible to build robust dialogue systems for real-world applications.
  • Leveraging Existing Resources: The strategy of aligning and fusing existing, high-quality TOD and CC datasets is an elegant and efficient way to create a new, valuable resource.
  • Model-Agnostic Framework: The CCET dataset and the proposed formalization are designed to be model-agnostic, meaning existing TOD models can be enhanced with CC capabilities without requiring a complete architectural overhaul.
  • Realism over Perfection: The authors argue that a slightly "noisy" or less perfectly curated dataset, like CCET, can actually be more beneficial for training models that need to perform in messy, real-world conversational environments.

Key Results

The paper primarily focuses on the construction of the CCET dataset and the methodology. While specific quantitative results from training models on CCET are not detailed in this excerpt, the authors state that:

  • The CCET dataset contains both TOD and CC utterances, mimicking real-life interaction environments.
  • It enables existing TOD models to gain generic chatting abilities with no annotation cost.
  • The automatic construction method is effective and affordable.

The paper also presents a statistical breakdown of the CCET dataset in Table II, showing a total of 500,000 training samples and 35,000 test samples across various fused components (T, C, TC, CT).

Limitations and Future Work

  • Quality vs. Human Data: The authors acknowledge that their automatically produced corpus might have lower quality compared to elaborately collected human data. However, they posit that this is acceptable and even beneficial for approximating real-world data distribution.
  • Filtering Rule for CT Fusion: The filtering rule for CT fusion (to ensure contextual relevance) is a heuristic and might discard potentially useful fused samples.
  • Evaluation Metrics: The paper proposes "probably available" metrics for fuse-motive systems, suggesting that standardized evaluation is still an open area.
  • Future Work: The authors hope their work will "propel future work on dialogue systems for fuse-motive tasks." This implies that further research into developing and evaluating models specifically designed for this fuse-motive paradigm is encouraged. The availability of the CCET dataset and code is intended to facilitate this.

Summary updated

Abstract

The goal of building intelligent dialogue systems has largely been separately pursued under two motives: task-oriented dialogue (TOD) systems, and open-domain systems for chit-chat (CC). Although previous TOD dialogue systems work well in the testing sets of benchmarks, they would lead to undesirable failure when being exposed to natural scenarios in practice, where user utterances can be of high motive-diversity that fusing both TOD and CC in multi-turn interaction. Since an industrial TOD system should be able to converse with the user between TOD and CC motives, constructing a fuse-motive dialogue dataset that contains both TOD or CC is important. Most prior work relies on crowd workers to collect and annotate large scale dataset and is restricted to English language setting. Our work, on the contrary, addresses this problem in a more effective way and releases a multi-turn dialogues dataset called CCET (Chinese Chat-Enhanced-Task). Meanwhile, we also propose a line of fuse-motive dialogues formalization approach, along with several evaluation metrics for TOD sessions that are integrated by CC utterances.
BibTeX
@misc{yu2022chit,
  author = {Changhong Yu and Chunhong Zhang and Qi Sun},
  title = {{A Chit-Chats Enhanced Task-Oriented Dialogue Corpora for Fuse-Motive Conversation Systems}},
  year = {2022},
  eprint = {2205.05886},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.05886},
}
Figure