Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization

Haode Zhang, Haowen Liang, Yuwei Zhang, Liming Zhan, Xiaolei Lu, Albert Y. S. Lam, Xiao-Ming Wu

TL;DR

Supervised pre-training (SPT) is a powerful tool for few-shot intent detection, but it often causes "representation degeneration," where sentence embeddings become cramped into a narrow cone (anisotropy). This paper introduces two simple yet effective regularizers—one based on contrastive learning and another on correlation matrices—that spread these embeddings out during training, significantly boosting accuracy on downstream few-shot tasks.

What Problem Does This Paper Address?

In task-oriented dialogue systems, identifying a user's intent (e.g., "transfer_money" vs. "check_balance") is critical. However, collecting thousands of labeled examples for every possible intent is impractical. Researchers have turned to Few-shot Intent Detection (FSID), where a model must learn to recognize new intents from just a handful of examples.

A common strategy is to take a Pre-trained Language Model (PLM) like BERT and fine-tune it on large, publicly available intent datasets before applying it to the specific target task. This is called Supervised Pre-training (SPT). While SPT helps the model learn "intent-ness," the authors discovered a major side effect: it makes the model's feature space highly anisotropic. Instead of using the full geometric space available, the model's embeddings huddle together in a narrow cone. This "representation degeneration" limits the model's ability to distinguish between subtle semantic differences, which is exactly what you need for high-performance intent detection.

Methods

The authors propose a joint training framework that balances the standard classification loss with a new isotropization loss. The goal is to learn intent features while forcing those features to spread out more evenly.

Overview of the proposed regularized supervised pre-training framework.

They introduce two specific regularizers:

  1. CL-Reg (Contrastive-learning-based): This uses a dropout-based approach. The same utterance is passed through the model twice with different dropout masks to create a "positive pair." Other utterances in the batch serve as "negative pairs." The loss pulls positive pairs together and pushes negative pairs apart, implicitly encouraging a more uniform (isotropic) distribution.
  2. Cor-Reg (Correlation-matrix-based): This is a more explicit geometric approach. It calculates the correlation matrix of the features in a batch and tries to minimize the difference between that matrix and the identity matrix. In essence, it tells the model: "Make sure the different dimensions of your feature vectors are not redundant."
Illustration of the CL-Reg and Cor-Reg mechanisms.

Datasets

The models were pre-trained on the OOS (Out-of-Scope) dataset, which contains diverse intents across 10 domains. For evaluation, the authors tested on three widely used benchmarks:

  • BANKING77: A specialized dataset with 77 intents in the banking domain.
  • HINT3: A dataset covering retail and gaming domains.
  • HWU64: A large-scale dataset with 64 intents across 21 different domains.

Key Insights and Ideas

  • Timing is Everything: The authors found that applying isotropization techniques (like whitening or contrastive learning) after fine-tuning often hurts performance. To be effective, the isotropization must happen during the supervised pre-training phase.
  • The "Goldilocks" Zone of Isotropy: More isotropy isn't always better. As shown in Figure 5, the best performance occurs when the feature space has a moderate degree of isotropy. If you force the embeddings to be perfectly spread out, you might destroy the semantic clusters the model needs to identify intents.
The relationship between isotropy and accuracy, showing a peak at moderate isotropy levels.
  • Correlation > Covariance: When designing Cor-Reg, the authors found that regularizing the correlation matrix is superior to the covariance matrix. This is because the correlation matrix is scale-invariant, making it easier to optimize without worrying about the varying magnitudes of different feature dimensions.

Key Results

The proposed regularized SPT consistently outperformed standard SPT (IntentBERT) and other state-of-the-art baselines across all datasets.

  • Accuracy Gains: On the BANKING77 dataset (5-way 2-shot), the BERT-based model improved from 80.38% (standard SPT) to 85.21% using the combined regularizers.
  • RoBERTa Performance: The gains were even more pronounced with RoBERTa, where the combined regularizers achieved 87.96% on BANKING77 2-shot, a significant jump over the baseline.
  • Efficiency: The computational overhead is negligible. As shown in Figure 7, the regularization terms take up only a tiny fraction of the total training time per epoch.

Limitations and Future Work

While the results are impressive, the authors note that the effectiveness of the regularizers depends on selecting the right weight ($\lambda$) for the regularization term, which currently requires a validation set.

Future work could explore:

  • Automated Weight Tuning: Developing methods to dynamically adjust the isotropization strength during training.
  • Broader NLU Application: Testing whether this joint training framework improves performance in other few-shot Natural Language Understanding (NLU) tasks, such as relation extraction or entity typing, which likely suffer from similar anisotropy issues.

Summary updated

Abstract

It is challenging to train a good intent classifier for a task-oriented dialogue system with only a few annotations. Recent studies have shown that fine-tuning pre-trained language models with a small amount of labeled utterances from public benchmarks in a supervised manner is extremely helpful. However, we find that supervised pre-training yields an anisotropic feature space, which may suppress the expressive power of the semantic representations. Inspired by recent research in isotropization, we propose to improve supervised pre-training by regularizing the feature space towards isotropy. We propose two regularizers based on contrastive learning and correlation matrix respectively, and demonstrate their effectiveness through extensive experiments. Our main finding is that it is promising to regularize supervised pre-training with isotropization to further improve the performance of few-shot intent detection. The source code can be found at https://github.com/fanolabs/isoIntentBert-main.
BibTeX
@inproceedings{zhang2022fine,
  author = {Haode Zhang and Haowen Liang and Yuwei Zhang and Liming Zhan and Xiaolei Lu and Albert Y. S. Lam and Xiao-Ming Wu},
  title = {{Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure