Topic Shift Detection in Chinese Dialogues: Corpus and Benchmark

Jiangyi Lin, Yaxin Fan, Feng Jiang, Xiaomin Chu, Peifeng LiarXiv 2023 · OpenReview

TL;DR

This paper introduces a new benchmark dataset for Chinese dialogue topic shift detection and proposes a novel teacher-student framework with hierarchical contrastive learning. This approach effectively addresses the challenge of predicting topic shifts without access to future dialogue turns, outperforming existing methods on both Chinese and English datasets.

What Problem Does This Paper Address?

Detecting when a conversation shifts topics is crucial for building more natural and engaging dialogue systems. This task, known as dialogue topic shift detection, can be broadly categorized into two types: "response-known" (where the next utterance is available) and "response-unknown" (where it's not). The latter is significantly more challenging because it requires predicting a shift based solely on past context. Existing research has largely focused on the response-known task, leaving a gap in understanding and modeling the response-unknown scenario, especially in non-English languages. Furthermore, there's a scarcity of high-quality, natural dialogue datasets for this task, particularly in Chinese. This paper aims to bridge these gaps by creating a new Chinese dataset and developing a robust model for the response-unknown topic shift detection.

Methods

The core of the proposed solution is a teacher-student framework designed to overcome the lack of response information.

  • Teacher-Student Architecture: A "teacher" model, which has access to both the dialogue context and the response, is trained first. A "student" model then learns from the teacher through knowledge distillation, aiming to mimic the teacher's ability to predict topic shifts using only the context. This distillation happens at multiple layers, not just the output, to better transfer knowledge.
  • Hierarchical Contrastive Learning: To further enhance the student's performance, a hierarchical contrastive learning approach is employed.
  • High-Level: Information Simulation Loss: This component focuses on learning a richer representation of the context by simulating the response information. It aims to align the context representations with those that would have been informed by the response, improving the student's predictive power.
  • Low-Level: Semantic Coherence-Aware Loss: This loss addresses the common issue of imbalanced datasets where topic shifts are less frequent than non-shifts. It uses contrastive learning to help the model better distinguish between different types of topic shifts, leading to more reliable predictions.

Datasets

  • Chinese Natural Topic Dialogue (CNTD) Corpus: The authors introduce a newly annotated corpus of 1308 Chinese dialogues, derived from the NaturalConv dataset. This corpus is designed to be high-quality and representative of natural conversations, filling a significant gap in existing Chinese dialogue topic datasets. It is split into training, validation, and test sets.
  • English TIAGE Corpus: This existing English dialogue topic corpus is used for evaluation to demonstrate the generalizability of the proposed model.

Key Insights and Ideas

  • The "Response-Unknown" Challenge is Real: The paper strongly emphasizes that predicting topic shifts without future information is a substantial challenge, and existing models struggle significantly.
  • Teacher-Student for Knowledge Transfer: The teacher-student paradigm is a powerful tool for imbuing a student model with knowledge it wouldn't otherwise have access to, especially when dealing with missing information like future dialogue turns.
  • Hierarchy Matters in Contrastive Learning: Applying contrastive learning at different levels (high-level for information simulation, low-level for label imbalance) proves more effective than a single approach.
  • Dataset Quality is Paramount: The creation of a high-quality, naturally annotated Chinese dataset (CNTD) is a critical contribution, enabling more robust research in this area.

Key Results

  • Improved Performance on CNTD: The proposed model significantly outperforms baseline models (T5, BERT) on the newly created CNTD dataset, achieving a 5.0% improvement in Micro-F1 score over the best baseline.
  • Effective Knowledge Distillation: The teacher-student framework (TS model) shows a 1.0% improvement over BERT, demonstrating the value of incorporating response information indirectly.
  • Hierarchical Contrastive Learning Boost: The full model (Ours) shows a further 4.0% improvement over the TS model, highlighting the effectiveness of the hierarchical contrastive learning components.
  • Generalizability: The model also achieves good results on the English TIAGE dataset, showing its applicability beyond Chinese dialogues.
  • Performance by Topic Count: The model performs better on dialogues with fewer topics, suggesting that longer, more complex conversations with numerous topic shifts remain a challenge.

Limitations and Future Work

  • Longer, Multi-Topic Dialogues: The paper acknowledges that detecting topic shifts in dialogues with many topics or very long turns remains a complex problem.
  • Real-time Detection: Future work will focus on improving the model's ability to detect topic shifts in real-time, potentially by better leveraging response information and news context.
  • Error Analysis: The authors note that errors often occur in "Introducing a relevant but different topic" and "commenting on the previous context" scenarios, suggesting these are particularly tricky for current models.

Summary updated

Abstract

Dialogue topic shift detection is to detect whether an ongoing topic has shifted or should shift in a dialogue, which can be divided into two categories, i.e., response-known task and response-unknown task. Currently, only a few investigated the latter, because it is still a challenge to predict the topic shift without the response information. In this paper, we first annotate a Chinese Natural Topic Dialogue (CNTD) corpus consisting of 1308 dialogues to fill the gap in the Chinese natural conversation topic corpus. And then we focus on the response-unknown task and propose a teacher-student framework based on hierarchical contrastive learning to predict the topic shift without the response. Specifically, the response at high-level teacher-student is introduced to build the contrastive learning between the response and the context, while the label contrastive learning is constructed at low-level student. The experimental results on our Chinese CNTD and English TIAGE show the effectiveness of our proposed model.
BibTeX
@misc{lin2023topic,
  author = {Jiangyi Lin and Yaxin Fan and Feng Jiang and Xiaomin Chu and Peifeng Li},
  title = {{Topic Shift Detection in Chinese Dialogues: Corpus and Benchmark}},
  year = {2023},
  eprint = {2305.01195},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.01195},
}
Figure