Effectiveness of Pre-training for Few-shot Intent Classification

Haode Zhang, Yuwei Zhang, Li-Ming Zhan, Jiaxin Chen, Guangyuan Shi, Albert Y. S. Lam, Xiao-Ming Wu

In the world of Natural Language Processing (NLP), the prevailing wisdom is often "bigger is better." To get a model to understand a specific task like intent classification—identifying what a user wants from a chatbot query—researchers typically take a massive language model like BERT and "continue" its pre-training on millions of unlabeled conversations.

However, a recent paper titled "Effectiveness of Pre-training for Few-shot Intent Classification" challenges this "brute force" approach. The authors demonstrate that a tiny amount of high-quality, labeled data is far more potent than a mountain of unlabeled text.

TL;DR

Instead of pre-training on millions of unlabeled dialogues, simply fine-tuning BERT on roughly 1,000 labeled examples from public datasets creates a powerful feature extractor called IntentBERT. This lightweight approach consistently outperforms massive, computationally expensive baselines in few-shot scenarios, even when the target domain is completely different from the training data.

What Problem Does This Paper Address?

Task-oriented dialogue systems (like those used for shopping or medical advice) need to accurately classify user "intents." The challenge arises during the early stages of development: you might only have 2 or 5 labeled examples per intent (the "few-shot" problem).

To bridge this gap, current state-of-the-art methods usually perform "continued pre-training" on massive unlabeled corpora (e.g., 700 million conversations). This is computationally expensive, slow, and assumes that having more data—even if it's unlabeled and noisy—is always the answer. The authors ask a provocative question: Can we instead leverage the "free lunch" of existing, high-quality public benchmarks to learn a transferable "intent-detecting" skill?

Methods

The authors propose two main strategies:

  1. Supervised Pre-training (IntentBERT): They take a standard BERT-base model and fine-tune it on a small, multi-domain labeled dataset (like OOS or HWU64). They simply add a linear classifier on top of the [CLS] token. Once trained, the classifier is discarded, and the tuned BERT serves as a fixed feature extractor for new, unseen tasks.
  2. Joint Pre-training: If unlabeled data from the target domain is available, they use a joint loss function. This combines the supervised classification loss from the source data with a Masked Language Modeling (MLM) loss on the target data. This helps the model adapt its semantic representations to the new domain while retaining the "intent structure" learned from the source.

Datasets

The study utilizes several key datasets to prove cross-domain generalization:

  • Source (Pre-training): OOS (120 intents across 10 domains) and HWU64 (64 intents across 21 domains).
  • Target (Evaluation): BANKING77 (fine-grained banking intents), MCID (Covid-19 queries), and HINT3 (Retail and Gaming).

Crucially, the authors show that the vocabulary overlap between these source and target datasets is very low (often around 10-15%), meaning the model cannot rely on simple word-matching to succeed.

Figure 2: Vocabulary overlap.

Key Insights and Ideas

  • The "Underlying Structure" of Intent: The most significant finding is that intent classification tasks across different domains (from banking to medical) likely share a similar underlying logical structure. This structure can be learned from just 1,000 labeled examples.
  • Efficiency over Scale: IntentBERT, trained on a tiny fraction of the data used by models like CONVBERT or TOD-BERT, achieves better results. This suggests that task-specific supervision is a much stronger signal than general domain adaptation via unlabeled text.
  • Saturation Points: The authors found that performance gains saturate quickly. Training on more than 4 domains or more than 1,000 total samples yields diminishing returns, making this a highly accessible method for developers with limited hardware.
Figure 3: Effect of the amount of labeled data used for pre-training in the source domain (OOS). The results are evaluated on 5-way 2-shot tasks on BANKING77.

Key Results

IntentBERT sets a new bar for few-shot intent classification:

  • Superior Accuracy: In 5-way 2-shot tasks on the MCID (Covid-19) dataset, IntentBERT (OOS) outperformed the strong CONVBERT baseline by an absolute margin of 9.4%.
  • Qualitative Superiority: Visualization using t-SNE shows that IntentBERT creates much tighter, more distinct clusters for different intents compared to standard BERT or even TOD-BERT, which was pre-trained on 100,000 dialogue samples.
Figure 1: Visualization of the embedding spaces with t-SNE. We randomly sample 10 classes and 500 data per class from BANKING77 (best viewed in color).
  • Joint Training Boost: When target unlabeled data is used, the "IntentBERT + MLM" approach provides an additional boost of up to 9.2% over the basic IntentBERT.

Limitations and Future Work

While highly effective, the paper notes a few areas for further exploration:

  • Model Scope: The current study focuses on BERT-base. The authors suggest that applying this joint pre-training scheme to other architectures like GPT-2 or ELMo is a promising next step.
  • Task Variety: The "underlying structure" hypothesis is proven here for intent classification, but it remains to be seen if similar "small-data supervised pre-training" works for other NLU tasks like named entity recognition (NER) or slot filling.

In conclusion, IntentBERT proves that high-quality supervision is a shortcut to generalization. For researchers and engineers, this means we can stop chasing massive datasets and start focusing on the "free lunch" already sitting in our public repositories.

Summary updated

Abstract

This paper investigates the effectiveness of pre-training for few-shot intent classification. While existing paradigms commonly further pre-train language models such as BERT on a vast amount of unlabeled corpus, we find it highly effective and efficient to simply fine-tune BERT with a small set of labeled utterances from public datasets. Specifically, fine-tuning BERT with roughly 1,000 labeled data yields a pre-trained model -- IntentBERT, which can easily surpass the performance of existing pre-trained models for few-shot intent classification on novel domains with very different semantics. The high effectiveness of IntentBERT confirms the feasibility and practicality of few-shot intent detection, and its high generalization ability across different domains suggests that intent classification tasks may share a similar underlying structure, which can be efficiently learned from a small set of labeled data. The source code can be found at https://github.com/hdzhang-code/IntentBERT.
BibTeX
@inproceedings{zhang2021effectiveness,
  author = {Haode Zhang and Yuwei Zhang and Li-Ming Zhan and Jiaxin Chen and Guangyuan Shi and Albert Y. S. Lam and Xiao-Ming Wu},
  title = {{Effectiveness of Pre-training for Few-shot Intent Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.96},
}
Figure