Strategies to Improve Few-shot Learning for Intent Classification and Slot-Filling

Samyadeep Basu, Amr Sharaf, Karine Ip Kiun Chong, Alex Fischer, Vishal Rohra, Michael Amoake, Hazem El-Hammamy, Ehi Nosakhare, Vijay Ramani, Benjamin Han

TL;DR

This paper tackles the challenge of training accurate intent classification and slot-filling models with very limited labeled data, a common issue in real-world NLU systems. They demonstrate that combining contrastive learning with existing meta-learning techniques significantly boosts performance, and explore how various data augmentation strategies can further improve these models, especially for intent classification.

What Problem Does This Paper Address?

Modern Natural Language Understanding (NLU) systems rely on intent classification (IC) and slot-filling (SF) to understand user requests. For example, in "Book me a table for 6 at Lebanese Taverna," "Restaurant Booking" is the intent, and "Party_Size: 6" and "Name: Lebanese Taverna" are slots. Training deep learning models for these tasks typically requires vast amounts of labeled data. However, in many real-world scenarios, especially during model deployment or for niche applications, obtaining such large datasets is impractical and expensive. This scarcity of data leads to the "few-shot learning" problem: how to build models that can generalize well from only a handful of examples per class. This paper focuses on improving few-shot IC and SF by investigating how to enhance existing meta-learning approaches, which are designed to learn from few examples.

Methods

The authors build upon prototypical networks, a popular meta-learning technique for few-shot learning. Their core contributions involve integrating two key strategies:

  • Contrastive Learning as a Regularizer: They introduce supervised contrastive losses to the meta-training process. Contrastive learning aims to pull similar data points closer together in an embedding space while pushing dissimilar ones apart. By applying this as a regularizer to the prototypical network's loss function, they encourage the model to learn more robust and generalizable representations for both intents and slots. They devise separate contrastive losses for intents (using the <cls> token embedding) and slots (using individual token embeddings).
  • Data Augmentation Strategies: They explore several data augmentation techniques to generate more training data, particularly for the few-shot setting. These include:
  • Slot-List Values: A novel approach that leverages the fact that some slot types have a finite set of possible values (e.g., "smoking room," "spa" for a "facility" slot). They create synthetic utterances by replacing existing slot values with others from the same list.
  • Backtranslation: Translating an utterance to another language and then back to the original language using a neural machine translation model.
  • EDA (Easy Data Augmentation): A set of simple text perturbation techniques including random insertion, deletion, swapping, and synonym replacement.

They investigate the impact of applying these augmentations at different stages of the meta-learning pipeline (e.g., during meta-training on support sets, or on both support and query sets).

Datasets

The paper evaluates their proposed methods on two standard NLU benchmarks for joint intent classification and slot-filling:

  • SNIPS: A more challenging dataset with intents from diverse domains.
  • ATIS: A dataset focused on the airline domain.

They follow standard episode construction techniques for creating few-shot splits, dynamically sampling classes and shots per class.

Key Insights and Ideas

  • Contrastive Learning is a Powerful Regularizer: The most significant finding is that incorporating contrastive losses as a regularizer during meta-training consistently improves few-shot intent classification accuracy and marginally improves slot-filling F1 scores. This suggests that contrastive learning helps create better, more discriminative prototypes.
  • Data Augmentation's Impact Varies by Stage: The effectiveness of data augmentation is highly dependent on when it's applied in the meta-learning pipeline. Augmenting data at different stages can yield varying improvements.
  • Slot-List Augmentation is Effective for Intents: The novel "slot-list values" augmentation strategy shows consistent gains in intent classification across different stages of meta-training.
  • Slot-Filling is More Challenging with Few Shots: Data augmentation provides less benefit for slot-filling compared to intent classification. The authors hypothesize this is due to the significantly lower number of examples per slot class in few-shot settings, making it harder for augmentation to have a substantial impact.
  • Combining Strategies Yields Best Results: The combination of contrastive learning and data augmentation (specifically slot-list values) during meta-training leads to the best overall performance, suggesting synergistic benefits. Contrastive learning helps create better prototypes, while data augmentation mitigates meta-overfitting.

Key Results

  • State-of-the-Art Performance: Their proposed approach, combining contrastive learning with prototypical networks, outperforms existing state-of-the-art methods for few-shot joint IC/SF tasks on both SNIPS and ATIS datasets.
  • Intent Accuracy Gains: Contrastive learning alone leads to noticeable improvements in intent classification accuracy.
  • Data Augmentation Impact: For SNIPS, data augmentation can lead to up to a 4% gain in IC accuracy (for Kmax=20) and 2% (for Kmax=100). EDA, when applied during both meta-training and meta-testing, shows a noteworthy gain.
  • Slot F1 Improvements are Modest: While contrastive learning offers some improvement for slot F1, data augmentation yields only limited gains, especially when the number of shots per slot is very low.

Limitations and Future Work

  • Slot-Filling Challenges: The paper acknowledges that data augmentation has a limited impact on slot-filling performance, particularly when the number of examples per slot class is very low. They suggest that future work could explore incorporating more explicit syntactic information (like POS tags or noun-phrases) to aid slot-filling.
  • Limited Augmentation Techniques: The authors focused on a limited set of NLP-specific augmentation techniques (EDA, backtranslation, slot-list values) due to their simplicity. They note that a wider range of augmentation techniques exists and that adapting them for meta-learning is a promising future research direction.
  • Hyperparameter Tuning: While they provide some details on hyperparameter settings, further exploration of hyperparameter sensitivity could be beneficial.
  • Future Directions: The paper suggests exploring more sophisticated meta-specific augmentation and contrastive learning techniques to further push the boundaries of few-shot NLU. They also propose incorporating syntactic information (POS tags, noun-phrases) as an additional feature or through multi-task learning to improve slot-filling performance.

Summary updated

Abstract

Intent classification (IC) and slot filling (SF) are two fundamental tasks in modern Natural Language Understanding (NLU) systems. Collecting and annotating large amounts of data to train deep learning models for such systems are not scalable. This problem can be addressed by learning from few examples using fast supervised meta-learning techniques such as prototypical networks. In this work, we systematically investigate how contrastive learning and data augmentation methods can benefit these existing meta-learning pipelines for jointly modelled IC/SF tasks. Through extensive experiments across standard IC/SF benchmarks (SNIPS and ATIS), we show that our proposed approaches outperform standard meta-learning methods: contrastive losses as a regularizer in conjunction with prototypical networks consistently outperform the existing state-of-the-art for both IC and SF tasks, while data augmentation strategies primarily improve few-shot IC by a significant margin
BibTeX
@inproceedings{basu2022strategies,
  author = {Samyadeep Basu and Amr Sharaf and Karine Ip Kiun Chong and Alex Fischer and Vishal Rohra and Michael Amoake and Hazem El-Hammamy and Ehi Nosakhare and Vijay Ramani and Benjamin Han},
  title = {{Strategies to Improve Few-shot Learning for Intent Classification and Slot-Filling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure