HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

Weixin Liang, Kaihui Liang, Zhou Yu

Abstract

Open-domain dialog systems have a user-centric goal: to provide humans with an engaging conversation experience. User engagement is one of the most important metrics for evaluating open-domain dialog systems, and could also be used as real-time feedback to benefit dialog policy learning. Existing work on detecting user disengagement typically requires hand-labeling many dialog samples. We propose HERALD, an efficient annotation framework that reframes the training data annotation process as a denoising problem. Specifically, instead of manually labeling training samples, we first use a set of labeling heuristics to label training samples automatically. We then denoise the weakly labeled data using the Shapley algorithm. Finally, we use the denoised data to train a user engagement detector. Our experiments show that HERALD improves annotation efficiency significantly and achieves 86% user disengagement detection accuracy in two dialog corpora.

Introduction

Evaluating open-domain dialog systems (chatbots) is notoriously difficult. Traditional metrics like BLEU—which measure how closely a bot's response matches a reference text—correlate poorly with actual human enjoyment. The gold standard is "user engagement," but training a model to detect when a user is getting bored or frustrated usually requires thousands of hand-labeled examples.

Manual annotation is a massive bottleneck: it is expensive, slow, and raises privacy concerns when dealing with sensitive conversation data. Furthermore, self-reported user ratings are often noisy and biased. The HERALD (Annotation Efficient Method) framework addresses these challenges by learning to detect disengagement accurately using only a small set of rules and a sophisticated denoising mechanism.

Methodology

The HERALD pipeline consists of two main stages designed to minimize human intervention while maintaining high data quality.

Overview of the HERALD two-stage framework: auto-labeling with heuristics followed by denoising with the Shapley algorithm.

Stage 1: Auto-labeling with Heuristics

Instead of manual labeling, the framework utilizes "labeling functions" based on Regular Expressions (Regex) and Dialog Acts (using the MIDAS framework). These heuristics identify four primary patterns of disengagement:

  • Complaints: e.g., "You already said that," "You're not listening."
  • Topic Dislike: e.g., "I don't like music," "This is boring."
  • Requests to Terminate: e.g., "Stop," "Bye," "Let's talk about something else."
  • Non-positive Responses: Ending a turn with "I don't know," "Maybe," or "Uh."

Stage 2: Denoising with the Shapley Algorithm

Because heuristic rules are imperfect, the resulting labels are "noisy." To address this, HERALD employs the Data Shapley algorithm. This algorithm quantifies the contribution of each training point to the model's performance. Data points with negative Shapley values—indicating they are likely mislabeled and detrimental to the model—are identified. Rather than discarding these points, HERALD "flips" their binary labels to correct them before final training of a BERT-based classifier.

Experimental Results

The framework was evaluated across two diverse conversation styles: the ASR-based Gunrock Movie Dataset (5,644 turns) and the text-based ConvAI2 Dataset (over 18,000 turns).

Performance comparison of HERALD against various heuristic and BERT-based baselines across Gunrock and ConvAI2 datasets.

As shown in the results table, HERALD achieved a balanced accuracy (bACC) of 86.17% on the Gunrock dataset and 86.22% on the ConvAI2 dataset. This significantly outperforms models trained only on small expert-labeled sets (~74%) or models using raw, un-denoised heuristics (~78%).

Key Insights

  • Quantity vs. Quality: Learning from a large volume of noisy, automatically-labeled data is more effective than learning from a very small set of perfect data, provided a robust denoising mechanism like Shapley is used.
  • The Power of the "Flip": Flipping binary labels instead of discarding noisy data preserves linguistic information while correcting the supervisory signal.
  • Generalizability: Heuristic rules developed for voice-based bots transferred well to text-based bots, suggesting universal patterns in how humans express disengagement.
  • Efficiency: Annotating the small development set required for the Shapley algorithm took less than one hour, a fraction of the time required for full manual dataset annotation.

Conclusion

HERALD demonstrates that combining weak supervision with algorithmic denoising can drastically reduce the manual labor required to train effective dialog evaluation models. Future work may explore more granular levels of engagement beyond binary classification and refine heuristics to better capture contextual nuances like sarcasm.

Summary updated

Abstract

Open-domain dialog systems have a user-centric goal: to provide humans with an engaging conversation experience. User engagement is one of the most important metrics for evaluating open-domain dialog systems, and could also be used as real-time feedback to benefit dialog policy learning. Existing work on detecting user disengagement typically requires hand-labeling many dialog samples. We propose HERALD, an efficient annotation framework that reframes the training data annotation process as a denoising problem. Specifically, instead of manually labeling training samples, we first use a set of labeling heuristics to label training samples automatically. We then denoise the weakly labeled data using the Shapley algorithm. Finally, we use the denoised data to train a user engagement detector. Our experiments show that HERALD improves annotation efficiency significantly and achieves 86% user disengagement detection accuracy in two dialog corpora.
BibTeX
@inproceedings{liang2021herald,
  author = {Weixin Liang and Kaihui Liang and Zhou Yu},
  title = {{HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=fB-8LuTETb1},
}
Figure