Label efficient semi-supervised conversational intent classification

Mandar Kulkarni, Kyung Kim, Nikesh Garera, Anusua TrivediACL 2023

TL;DR

This paper tackles the challenge of building effective chatbots for e-commerce by improving intent classification, especially for queries with code-mixing and grammatical errors. They propose a novel semi-supervised learning (SSL) approach that leverages a small amount of labeled data with a large pool of unlabeled queries, achieving significant accuracy gains over purely supervised methods.

What Problem Does This Paper Address?

In e-commerce, chatbots are crucial for providing a seamless user experience by understanding and responding to a wide range of customer queries, from pre-purchase inquiries about product details to post-purchase issues like returns. A core component of these chatbots is intent classification – accurately identifying what a user wants. However, this task is complicated by the diverse user base, leading to a significant percentage of queries that are in "code-mix" (e.g., Hinglish) or contain grammatical errors. Training a robust intent classifier typically requires a massive amount of manually labeled data, which is expensive and time-consuming to obtain. This paper addresses the need for a label-efficient method that can achieve high accuracy even with limited labeled data, by effectively utilizing readily available unlabeled query data.

Methods

The authors propose a semi-supervised learning (SSL) approach that combines supervised and unsupervised learning techniques. For supervised learning on a small labeled dataset, they explore data augmentation strategies:

  • MixUp: A technique that creates new training samples by linearly interpolating existing samples and their labels in the representation space.
  • Grammar Augmentation: Introducing spell errors and word swaps to create more diverse training data.

For unsupervised learning on a larger unlabeled dataset, they employ label consistency with dropout noise. This method involves passing unlabeled queries through the model twice with dropout enabled, and then enforcing that the model's predictions for these two passes are consistent. This encourages the model to make robust predictions that are less sensitive to small perturbations. The overall model is trained by combining supervised and unsupervised loss functions. They experiment with pre-trained transformer models like BERT and Sentence-BERT.

Datasets

  • Labeled Data: A dataset of 7,000 manually labeled query samples, with 250 samples per intent across 28 predefined intents.
  • Test Data: A manually tagged dataset of 7,569 samples, which includes code-mix Hinglish queries and grammatically incorrect queries.
  • Unlabeled Data: A large corpus of approximately 925,000 unlabeled queries from an internal database.

The paper notes that all queries (labeled and unlabeled) are converted to lowercase and punctuation is removed.

Key Insights and Ideas

  • Sentence-BERT shines with limited data: Sentence-BERT models consistently outperform standard BERT models, especially when only a small number of labeled examples are available. This highlights its effectiveness for few-shot learning scenarios.
  • Unsupervised consistency is key: The proposed unsupervised label consistency loss with dropout noise provides a significant advantage over purely supervised baselines, even when training is halted based on converged training loss rather than a validation set.
  • Dropout is a robust regularizer: Interestingly, training with dropout label consistency appears to be less prone to overfitting, even when training continues until convergence without a validation set.
  • MSE loss for consistency is superior: When using dropout for label consistency, Mean-Square-Error (MSE) loss consistently outperforms Cross-Entropy (CE) loss for measuring the difference between predictions.
  • Dropout probability has minimal impact: Varying the dropout probability within a reasonable range (0.05 to 0.2) does not significantly affect the model's accuracy, suggesting the robustness of the approach.

Key Results

  • The proposed semi-supervised approach significantly improves intent classification accuracy over supervised baselines, even with a limited labeled dataset.
  • Sentence-BERT models achieve better calibration (lower Expected Calibration Error) compared to BERT models.
  • The model trained with supervised data augmentation (grammar + MixUp) shows only a slight advantage over the purely supervised baseline, which the authors attribute to potential overfitting due to the lack of a validation set for early stopping. However, when combined with unsupervised consistency loss, significant gains are observed.
  • A variant of the proposed model is currently deployed in production, indicating its practical utility.

Limitations and Future Work

  • Overfitting with Supervised MixUp alone: The paper suggests that supervised MixUp, when trained without validation monitoring and with a small labeled corpus, can be prone to overfitting. This indicates that unsupervised consistency is crucial for robust performance in low-resource settings.
  • Validation Set Impact: The authors hypothesize that the limited improvement from supervised MixUp alone is due to the lack of a validation set for early stopping. An ablation study confirms this, showing that using a validation set improves performance for supervised MixUp.
  • Future Work: While not explicitly detailed, the success of the deployed model suggests potential for further refinement and expansion of the SSL techniques for conversational AI tasks. The paper focuses on BERT and Sentence-BERT, leaving room to explore other transformer architectures.

Summary updated

BibTeX
@inproceedings{kulkarni2023label,
  author = {Mandar Kulkarni and Kyung Kim and Nikesh Garera and Anusua Trivedi},
  title = {{Label efficient semi-supervised conversational intent classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure