HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

Weixin Liang, Kaihui Liang, Zhou Yu

TL;DR

This paper introduces HERALD, an efficient framework for automatically detecting user disengagement in social conversations. By reframing annotation as a denoising problem and leveraging Shapley values, HERALD significantly reduces the cost of creating training data while achieving high accuracy in identifying disengaged users.

What Problem Does This Paper Address?

The ultimate goal of open-domain dialogue systems is to provide an enjoyable user experience. A key metric for this is user engagement, which can also serve as real-time feedback for improving dialogue policy learning. However, accurately detecting user disengagement is challenging. Existing methods often require extensive manual annotation of dialogue samples, which is time-consuming and expensive, especially for privacy-sensitive data. Furthermore, traditional automatic evaluation metrics like BLEU correlate poorly with human judgment, and self-reported user ratings are prone to bias and variance. This paper aims to address the bottleneck of data annotation for user disengagement detection.

Methods

HERALD employs a two-stage pipeline to efficiently annotate training data and build a user engagement detector.

Stage 1: Auto-labeling with Heuristic Functions: Instead of manual labeling, HERALD uses a set of heuristic functions to automatically label training data. These heuristics are based on identified patterns of user disengagement, categorized into four groups: complaining about system responses, disliking the current topic, requesting to end or change topics, and ending with non-positive responses. These heuristics are implemented using regular expressions (Regexes) and can be augmented with NLU services.

Stage 2: Denoising with Shapley Algorithm: Since the automatically generated labels are noisy, HERALD uses the Shapley algorithm to denoise the weakly labeled data. The Shapley algorithm quantifies the contribution of each training datum to the model's performance. Data points with negative Shapley values are identified as potentially mislabeled and their labels are corrected by flipping them. This denoising process is made computationally feasible by reducing the deep model to a K-nearest neighbors (KNN) classifier in the feature space.

Finally, the denoised data is used to fine-tune a BERT-based model to obtain the final user engagement detector.

Datasets

The paper uses several open-domain dialogue datasets for training and evaluation:

  • Gunrock Movie Dataset: An ASR-based social chatbot dataset collected from an in-lab user study.
  • ConvAI2 Dataset: A text-based dataset from the second Conversational Intelligence Challenge.
  • Google Meena Dataset: Used for studying example Human-Meena conversations.
  • Facebook Blender Dataset: Used for studying example Human-Blender conversations.

The authors also highlight that their approach is generalizable to various corpora.

Key Insights and Ideas

  • Reframing Annotation as Denoising: The core innovation is treating data annotation as a denoising problem, moving away from purely manual labeling.
  • Shapley Values for Data Valuation: Using Shapley values provides a principled way to identify and correct noisy labels by quantifying the contribution of each data point.
  • Heuristics as a Scalable Weak Labeler: Heuristic functions, especially when combined with Regexes and NLU, offer a cost-effective way to generate initial labels for large datasets.
  • Noisy Data Can Be Beneficial: Learning from a large number of noisy labels (BERT(Auto)) can outperform learning from a limited number of clean labels (BERT(dev)).
  • Heuristics Generalize Across Modalities: The developed heuristic functions show effectiveness on both ASR-based and text-based dialogue systems.

Key Results

HERALD achieves 86% user disengagement detection accuracy on two dialogue corpora (Gunrock Movie and ConvAI2).

  • The heuristic labeling alone achieves decent performance, outperforming a BERT-based model trained on limited samples.
  • BERT fine-tuned on automatically labeled data (BERT(Auto)) significantly outperforms BERT fine-tuned on clean but small development sets.
  • The Shapley denoising step significantly improves performance, leading to HERALD's high accuracy.
  • Removing data points with low Shapley values consistently improves performance, demonstrating the effectiveness of the denoising strategy.
  • The annotation cost for the development set used in Shapley denoising is minimal (less than 1 hour).

Limitations and Future Work

  • ASR Errors: The paper acknowledges that ASR errors can lead to misinterpretations and incorrect auto-labeling, as shown in an error analysis example (Figure 3).
  • Heuristic Coverage: While comprehensive, heuristics might not cover all nuanced forms of disengagement, as illustrated in Figure 5 where a user's indirect request to change topic was missed.
  • Future Work: The authors suggest exploring alternative data valuation methods (like influence functions and TracIn, which showed promising results but slightly lower than HERALD) and extending HERALD to other machine learning tasks. They also mention improving the BERT encoder for multi-turn dialogs and handling imbalanced classes more effectively.

Summary updated

BibTeX
@inproceedings{liang2021herald,
  author = {Weixin Liang and Kaihui Liang and Zhou Yu},
  title = {{HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=fB-8LuTETb1},
}
Figure