The First Evaluation of Chinese Human-Computer Dialogue Technology

Weinan Zhang, Zhigang Chen, Wanxiang Che, Guoping Hu, Ting LiuarXiv 2017 · OpenReview

Introduction

Human-computer dialogue systems, ranging from virtual assistants like Siri and Alexa to open-domain "chit-chat" bots, have seen a surge in popularity. However, evaluating these systems remains a significant challenge, particularly for the Chinese language, which lacks a unified evaluation standard. This paper introduces the first standardized evaluation framework for Chinese human-computer dialogue systems, providing a benchmark and dataset sourced from iFLYTEK to measure progress in the field.

The authors identify a fundamental divide in conversational AI:

  1. Open-domain chit-chat: Systems designed for long, natural conversations without a specific goal. These are difficult to evaluate because there is no single "correct" response.
  2. Task-oriented systems: Systems designed to help users achieve specific goals, such as booking a flight or checking the weather.
Comparison between Open Domain Chit-Chat and Task-oriented Dialogue systems.

Evaluation Methodology

The proposed framework consists of two primary tasks designed to reflect real-world industrial challenges.

Task 1: User Intent Classification

Intent classification serves as the "front door" of any dialogue system. In this task, systems must classify a single user utterance into one of 31 categories, such as "booking air tickets" or "inquiring about weather." The evaluation was split into a Closed Evaluation (using only provided data) and an Open Evaluation (allowing external data) to assess the impact of external knowledge and pre-training.

Task 2: Online Testing of Task-oriented Dialogue

This task involves live interaction between human testers and the participant systems. Testers are given specific goals—for instance, "Inquire about a soft berth ticket from Harbin to Beijing for tomorrow evening"—and must interact with the system to complete the request.

Datasets and Knowledge Base

The evaluation utilized data provided by iFLYTEK Corporation, derived from real-world applications.

  • Task 1 Dataset: 4,000 labeled utterances covering diverse intents like music, travel, and apps.
  • Task 2 Dataset: 11 complete user intent examples and three JSON data files containing one month's worth of flight, hotel, and train information to serve as the system's knowledge base.

Key Insights and Metrics

The paper emphasizes several critical aspects of dialogue system performance:

  • The Intent Bottleneck: Accurate intent classification is vital; failure here leads to total pipeline failure.
  • Efficiency and Penalty Rules: To prioritize efficiency, task-oriented systems were penalized if they exceeded 30 turns, marking the interaction as a failure.
  • Multi-Dimensional Metrics: Beyond simple accuracy, Task 2 employed metrics such as Task Completion Ratio, User Satisfaction, Response Fluency, Number of Dialogue Turns, and Guidance Ability (handling out-of-database inputs).

Results

In Task 1, the Spoken Dialogue System Lab at South China Agricultural University achieved the highest scores in both tracks (0.9391 F1 for closed, 0.9414 F1 for open). The minimal difference suggests the provided dataset was highly robust.

Task 2 proved significantly more challenging. Only four out of seven submitted systems functioned correctly during live testing. While iDeepMind Artificial Intelligence achieved the highest task completion ratio (0.3175), the system from Beijing University of Posts and Telecommunications scored higher in user satisfaction (78.72). This discrepancy highlights the complex relationship between task completion and overall user experience.

Limitations and Future Work

The authors acknowledge limitations, including the small sample size for Task 2 (11 intent examples) and the lack of robustness in many research models. Future work will focus on expanding the evaluation to include more rigorous automated methods for assessing the "human-likeness" and fluency of open-domain Chinese conversations.

Summary updated

Abstract

In this paper, we introduce the first evaluation of Chinese human-computer dialogue technology. We detail the evaluation scheme, tasks, metrics and how to collect and annotate the data for training, developing and test. The evaluation includes two tasks, namely user intent classification and online testing of task-oriented dialogue. To consider the different sources of the data for training and developing, the first task can also be divided into two sub tasks. Both the two tasks are coming from the real problems when using the applications developed by industry. The evaluation data is provided by the iFLYTEK Corporation. Meanwhile, in this paper, we publish the evaluation results to present the current performance of the participants in the two tasks of Chinese human-computer dialogue technology. Moreover, we analyze the existing problems of human-computer dialogue as well as the evaluation scheme itself.
BibTeX
@misc{zhang2017first,
  author = {Weinan Zhang and Zhigang Chen and Wanxiang Che and Guoping Hu and Ting Liu},
  title = {{The First Evaluation of Chinese Human-Computer Dialogue Technology}},
  year = {2017},
  eprint = {1709.10217},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.10217},
}
Figure