Semi-supervised Meta-learning for Cross-domain Few-shot Intent Classification

Yue Li, Jiong ZhangACL 2021

Sure, here's a blog-style summary of the paper "Semi-supervised Meta-learning for Cross-domain Few-shot Intent Classification":

TL;DR

This paper introduces a novel semi-supervised meta-learning approach that combines the power of pre-trained language models (like BERT) with metric-based learning (ProtoNet) to improve few-shot intent classification. By leveraging unlabeled data through self-supervision, the method achieves better generalization to new domains and improved detection of out-of-scope intents compared to purely supervised meta-learning.

What Problem Does This Paper Address?

The paper tackles the challenge of few-shot intent classification, a scenario where models need to learn to classify user intents with very limited labeled examples. Traditional deep learning models require large amounts of labeled data, which is often scarce for new tasks or domains. Meta-learning offers a promising solution by training models to "learn to learn" from a variety of tasks, enabling rapid adaptation to new ones. However, existing meta-learning approaches often rely on a diverse set of labeled meta-training tasks, which can be difficult and expensive to create. This paper addresses the gap of how to effectively perform meta-learning in a data-efficient, semi-supervised manner, particularly for cross-domain scenarios where the model must generalize to intents from entirely new domains.

Methods

The core of the proposed method is a combination of two key components:

  • Pre-trained Language Model (BERT/RoBERTa) as an Embedding Function: The authors leverage large pre-trained language models (PLMs) like BERT and RoBERTa as the encoder. These PLMs provide powerful, general-purpose representations that are then fine-tuned during meta-training to become task-agnostic. This allows the model to benefit from the rich linguistic knowledge embedded in these models.
  • Prototypical Networks (ProtoNet) for Classification: ProtoNet is a metric-based meta-learning algorithm. It works by learning to map input samples into an embedding space where class prototypes (average embeddings of support examples for each class) can be computed. New query examples are then classified based on their distance to these prototypes. ProtoNet is chosen for its efficiency, scalability, and simpler inductive bias, making it suitable for few-shot scenarios.

The novelty lies in the semi-supervised meta-training approach. Instead of relying solely on labeled meta-training tasks, the authors incorporate Subset Masked Language Modeling Tasks (SMLMT). SMLMT is a self-supervised technique where tokens are masked in sentences, and the model learns to predict the masked token. By training ProtoNet on both supervised intent classification tasks and these SMLMT tasks derived from unlabeled data, the model learns a more robust and generalizable representation. This semi-supervised approach aims to prevent overfitting on small labeled datasets and improve cross-domain adaptation.

Datasets

The primary dataset used for evaluation is CLINC150. This dataset is designed for intent classification and features 150 different intent classes spread across 10 domains. The paper utilizes various splits of CLINC150 to evaluate different few-shot learning settings, including:

  • Single domain unseen examples
  • Multi-domain unseen examples
  • Multi-domain unseen classes
  • Multi-domain unseen domain

The paper also mentions using unlabeled data from the same domain for the SMLMT tasks.

Key Insights and Ideas

  • Self-supervision bridges the gap: The most significant insight is that self-supervised tasks like SMLMT can effectively augment limited labeled data in meta-training, leading to substantial improvements in few-shot generalization. This is crucial for making meta-learning practical in real-world scenarios where labeled data is scarce.
  • ProtoNet + SMLMT outperforms supervised meta-training: The semi-supervised approach (ProtoNet + SMLMT) consistently outperforms purely supervised meta-training, even when the supervised approach uses more labeled data. This highlights the power of leveraging unlabeled data.
  • Pre-trained models are essential: The choice of pre-trained language model significantly impacts performance. RoBERTa, as an embedding function, generally yields better results than BERT, especially in challenging scenarios like out-of-scope detection.
  • Out-of-scope detection benefits from meta-training: Meta-training, particularly the semi-supervised variant, significantly improves the model's ability to distinguish between in-domain intents and out-of-scope utterances.
  • Token importance shifts with meta-training: Visualizing token importance reveals that meta-training helps the model focus on more relevant words for intent classification, leading to better understanding and adaptation.

Key Results

  • Improved Few-Shot Accuracy: The semi-supervised ProtoNet + SMLMT approach achieved significantly higher meta-test accuracy. For example, with K=10 shots, it reached 93.9% accuracy, outperforming supervised ProtoNet (90.6%) and even supervised ProtoNet with 150 labeled examples (86.4% from Table 3).
  • Enhanced Out-of-Scope (OOS) Detection: The semi-supervised approach also led to the best OOS F1 scores, with 0.601 for BERT and 0.766 for RoBERTa. This demonstrates a substantial improvement in identifying irrelevant utterances.
  • Effect of K-shots: Increasing the number of support examples (K) per task generally improves accuracy, with performance plateauing around K=20.
  • RoBERTa's Advantage: Using RoBERTa as the embedding function consistently yielded better results than BERT, showing a nearly 10-point improvement in in-domain accuracy and OOS F1 score after meta-training.

Limitations and Future Work

  • Limitations of ProtoNet: The paper acknowledges that ProtoNet has simpler inductive biases, which might limit its performance compared to more complex meta-learners.
  • Future Directions: The authors suggest exploring:
  • Combining various types of meta-learners.
  • Designing different self-supervised tasks.
  • Validating the algorithms on other datasets.
  • Extending the approach beyond classification problems.

Summary updated

BibTeX
@inproceedings{li2021semi,
  author = {Yue Li and Jiong Zhang},
  title = {{Semi-supervised Meta-learning for Cross-domain Few-shot Intent Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure