A Semi-supervised Multi-task Learning Approach to Classify Customer Contact Intents

Li Dong, Matthew C. Spencer, Amir BiagiACL 2021

TL;DR

This paper tackles the challenge of accurately classifying customer intents in e-commerce support, a crucial step for efficient routing and response. They propose a novel semi-supervised, multi-task learning approach that leverages both confirmed positive and challenging negative customer feedback, along with unlabeled data, to significantly improve intent classification accuracy.

What Problem Does This Paper Address?

In e-commerce customer support, understanding the intent behind a customer's contact (e.g., "how to set up an account," "inquiry about a product") is vital for routing them to the right agent or providing an automated response. Traditional machine learning approaches often struggle with the dynamic nature of intent taxonomies, the cost of manual labeling, and the presence of "hard negative" cases – instances that are confidently misclassified by the model. This paper addresses the specific challenge of building robust intent classifiers when dealing with a data composition of confirmed positive cases, unlabeled data, and difficult-to-label negative cases, which are common in real-world customer support scenarios.

Methods

The core of the proposed solution is a Semi-supervised Multi-task Learning (SS MT) approach combined with Domain and Task Adaptive Pretraining (D/TAPT) of the ALBERT language model.

Here's a breakdown:

  • ALBERT Backbone: They utilize ALBERT (A Lite BERT), a more parameter-efficient variant of BERT, as their base language model.
  • Domain and Task Adaptive Pretraining (DAPT/TAPT): Before fine-tuning for the specific intent classification task, they further pre-train ALBERT on a large corpus of customer contact data (DAPT) and then on the specific task data (TAPT). This helps the model adapt to the nuances of customer support language.
  • Multi-task Learning (MT): Instead of a single multiclass classification, they frame the problem as multiple tasks:
  • A primary multiclass classification task for all identified intents.
  • Multiple binary classification tasks, one for each intent, where "positive" cases are confirmed intents and "negative" cases are the challenging, misclassified instances identified by customer support representatives (CSRs). This allows the model to learn from these difficult examples.
  • Semi-supervised Learning (SS): To leverage the large volume of unlabeled customer contact data, they employ an iterative approach. The model makes predictions on unlabeled data, and high-confidence predictions are then used to augment the training set for the multiclass classification task.
  • End-to-End Training: All these tasks are trained jointly in an end-to-end fashion, sharing the ALBERT encoder.

Datasets

The paper uses a proprietary dataset from an e-commerce website.

  • Curated Data: This includes positive cases with confirmed intents and negative cases identified by CSRs. The data is unbalanced across classes, with a positive-to-negative ratio of approximately 4:1 for each class.
  • Un-curated Data: This constitutes a significantly larger portion of the data (roughly 20 times the curated data) and consists of unlabeled customer contacts.

No new benchmarks are introduced; the focus is on applying existing techniques to a real-world customer support dataset.

Key Insights and Ideas

  • Leveraging "Hard Negatives": The most significant insight is the value of incorporating "hard negative" cases into the training process. These are instances that the model confidently misclassifies, and learning from them dramatically improves robustness.
  • Multi-task Learning for Efficiency: Framing intent classification as a multi-task problem (multiclass + binary for negatives) is an elegant way to utilize all available signals without requiring extensive manual relabeling of negative cases.
  • Adaptive Pretraining is Crucial: Pretraining ALBERT on domain-specific customer contact data (DAPT/TAPT) significantly improves the quality of the learned representations, leading to better clustering and classification.
  • Semi-Supervision for Scale: The iterative semi-supervised approach effectively utilizes the vast amount of unlabeled data, further boosting performance without additional human annotation costs.

Key Results

The proposed SS MT D/TAPT ALBERT model achieved substantial improvements over a baseline finetuned ALBERT model:

  • Average AUC ROC: Increased by 19.9 points.
  • Sample-weighted average AUC ROC: Increased by 17.0 points.

Incremental analysis showed that the Multi-task (MT) strategy alone provided a significant boost (17.8 points average AUC ROC), with DAPT, TAPT, and SS each adding roughly 1-2 points on top.

Limitations and Future Work

  • Language Specificity: The current study is limited to English. The authors suggest exploring cross-lingual language models for low-resource languages.
  • Feature Integration: While they use text features, they acknowledge that incorporating structured customer interaction features could further enhance performance.
  • Hierarchical Taxonomies: The current approach is for flat taxonomies. Future work could extend this to handle more complex, hierarchical intent structures.
  • Feedback Granularity: The current feedback is binary ("yes"/"no"). Investigating how to incorporate more nuanced feedback could be beneficial.
  • Computational Cost: Adaptive pretraining and semi-supervised learning are computationally intensive. While MT is cost-effective, the overall training pipeline requires significant resources.
  • Batching Strategy: The paper notes that training each task in a separate batch might not be optimal, and exploring joint batching could be a future research direction.

Summary updated

BibTeX
@inproceedings{dong2021semi,
  author = {Li Dong and Matthew C. Spencer and Amir Biagi},
  title = {{A Semi-supervised Multi-task Learning Approach to Classify Customer Contact Intents}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure