Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning

Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, Philip Yu

TL;DR

This paper tackles the challenging problem of few-shot intent detection, where systems must accurately identify user intents with very few examples, especially when intents are fine-grained and semantically similar. They propose a novel two-stage approach: self-supervised contrastive pre-training to learn general utterance discrimination, followed by supervised contrastive fine-tuning to explicitly distinguish similar intents. This method achieves state-of-the-art results on several challenging datasets.

What Problem Does This Paper Address?

Intent detection is a crucial component of task-oriented dialogue systems, enabling them to understand what a user wants to do. However, in real-world applications, new intents emerge frequently, and collecting a large number of labeled examples for each is expensive and time-consuming. This leads to the "few-shot" intent detection problem, where the system must learn to classify intents with only a handful of examples (e.g., 5 or 10 per intent). This challenge is exacerbated when intents are "fine-grained" and semantically similar, making it difficult for models to differentiate between them, even with more data. Existing methods often rely on data augmentation or task-adaptive training with pre-trained models, but these can be inefficient or struggle with the nuances of highly similar intents.

Methods

The proposed method, Contrastive Pre-Training and Fine-Tuning (CPFT), consists of two main stages:

  1. Self-supervised Contrastive Pre-training:
    • The authors leverage contrastive learning, a technique that learns to pull similar data points closer and push dissimilar ones apart in an embedding space.
  2. In this stage, they pre-train an encoder (BERT-based) on a large collection of intent utterances without using any intent labels.
  3. The goal here is to implicitly learn to discriminate semantically similar utterances, building a foundational understanding of language nuances.
  4. They also incorporate a mask language modeling (MLM) loss to enhance token-level understanding.
  5. Supervised Fine-tuning:
    • After pre-training, the model is fine-tuned on the specific few-shot intent detection task.
  6. This stage combines two losses:
  7. Intent Classification Loss: The standard loss for classifying utterances into their respective intents.
  8. Supervised Contrastive Learning Loss: This explicitly uses the limited labeled data to further pull utterances from the same intent closer and push utterances from different intents further apart. This is crucial for distinguishing semantically similar intents.
  9. The authors note that even an utterance and its augmented version (e.g., with masked tokens) can form a positive pair due to the inherent randomness of dropout in the encoder.

Datasets

  • Pre-training Datasets: Six public datasets were collected and combined for the self-supervised pre-training phase. The paper mentions excluding test sets and utterances with fewer than five tokens, resulting in 80,782 training utterances.
  • Evaluation Datasets: Three challenging datasets were used for evaluation, known for their fine-grained and similar intents:
  • CLINC150: 150 intents across 10 domains.
  • BANKING77: 77 intents within a single domain.
  • HWU64: 64 intents across 21 domains. The authors followed standard few-shot evaluation protocols, using 5-shot and 10-shot settings.

Key Insights and Ideas

  • Contrastive Learning for Implicit Discrimination: The self-supervised contrastive pre-training stage is key to learning general semantic discrimination without explicit labels, which is crucial for handling the vastness of language.
  • Supervised Contrastive Learning for Explicit Discrimination: The supervised contrastive loss during fine-tuning is vital for explicitly pushing apart semantically similar intents, a common challenge in few-shot settings.
  • Two-Stage Approach is Necessary: Ablation studies show that both the self-supervised pre-training and supervised fine-tuning stages are essential for achieving optimal performance. Removing either stage leads to significant performance drops.
  • Pre-training on Target Data is Beneficial: Including the target intent dataset in the pre-training phase yields better results than pre-training on external data alone.
  • Joint Training vs. Two-Stage: While joint training of both losses is possible, the proposed sequential two-stage approach is more efficient and requires less retraining for new settings.

Key Results

  • State-of-the-Art Performance: CPFT achieves state-of-the-art results on all three challenging datasets (CLINC150, BANKING77, HWU64) under both 5-shot and 10-shot settings.
  • Outperforms Strong Baselines: CPFT outperforms strong baselines like DNNC (a discriminative nearest-neighbor model with data augmentation) by significant margins (e.g., 1.32% and 1.57% on CLINC150 and HWU64 in 5-shot).
  • Improved Variance: CPFT exhibits lower variance in performance compared to DNNC, indicating more stable and reliable results.
  • Efficiency: The pre-training takes about 2.5 hours, and fine-tuning is very fast (around 5 minutes per setting), making it efficient for adapting to new tasks or intents. This is a significant advantage over methods like DNNC that require extensive retraining.

Limitations and Future Work

  • Hyper-parameter Sensitivity: The authors note that the supervised contrastive learning loss can be sensitive to hyper-parameters during fine-tuning, especially with very limited training examples.
  • Future Work: The paper suggests exploring whether self-supervised contrastive pre-training only on the target intent dataset could be beneficial, leaving this as a future research direction. They also mention the potential for further investigation into the impact of batch size and other hyper-parameters.

Summary updated

Abstract

In this work, we focus on a more challenging few-shot intent detection scenario where many intents are fine-grained and semantically similar. We present a simple yet effective few-shot intent detection schema via contrastive pre-training and fine-tuning. Specifically, we first conduct self-supervised contrastive pre-training on collected intent datasets, which implicitly learns to discriminate semantically similar utterances without using any labels. We then perform few-shot intent detection together with supervised contrastive learning, which explicitly pulls utterances from the same intent closer and pushes utterances across different intents farther. Experimental results show that our proposed method achieves state-of-the-art performance on three challenging intent detection datasets under 5-shot and 10-shot settings.
BibTeX
@inproceedings{zhang2021few,
  author = {Jianguo Zhang and Trung Bui and Seunghyun Yoon and Xiang Chen and Zhiwei Liu and Congying Xia and Quan Hung Tran and Walter Chang and Philip Yu},
  title = {{Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.144},
}
Figure