Diversity Features Enhanced Prototypical Network for Few-shot Intent Detection

Fengyi Yang, Xi Zhou, Yi Wang, Abibulla Atawulla, Ran BiIJCAI 2022 · DOI

TL;DR

This paper introduces DFEPN, a novel method for few-shot intent detection (FSID) and generalized few-shot intent detection (GFSID) in dialogue systems. DFEPN enhances intent representations by transferring "diversity features" from abundant "seen" intent classes to scarce "novel" intent classes, leading to improved accuracy on both FSID and GFSID tasks.

What Problem Does This Paper Address?

Intent Detection (ID) is crucial for dialogue systems, enabling them to understand user requests. However, building ID models for new dialogue systems is challenging due to the scarcity of labeled data, a problem known as Few-Shot Intent Detection (FSID). Existing FSID methods struggle with Generalized Few-Shot Intent Detection (GFSID), where the model must classify intents from both previously seen and entirely new categories. A core issue is that limited training samples fail to capture the full diversity of user expressions for any given intent. This paper aims to address this by effectively leveraging the diversity of existing, well-represented intents to improve the understanding of new, sparsely represented intents.

Methods

The proposed method, Diversity Features Enhanced Prototypical Network (DFEPN), builds upon the Prototypical Network framework. Its key components are:

  • Semantic Encoder (SE): Utilizes a pre-trained BERT model, further fine-tuned on the training data, to encode utterances into high-dimensional semantic representations. This captures rich linguistic information.
  • Diversity Feature Generator (DFG): This is the core innovation. It learns to extract "diversity features" by analyzing the differences between pairs of samples from a "source domain" (intents with abundant data). These diversity features are then transferred to "target domain" (novel intents) by adding them to the target intent vectors. This process aims to enrich the representation of novel intents with the expressive variations seen in other classes.
  • Feature Fusion (FF): This module intelligently fuses the original support vectors with the generated diversity-enhanced vectors. It uses a residual connection and a linear layer to create a more robust prototype vector for each class.
  • Similarity Classifier: Employs Euclidean distance to measure the similarity between query set vectors and the learned prototype vectors, determining the final classification score.

DFEPN also incorporates a meta-learning framework to construct numerous training episodes, simulating the few-shot scenario and enabling the model to learn from limited data. Notably, both support and query set vectors are enhanced using the DFG and FF modules, allowing the model to capture interactive knowledge between them.

Datasets

The paper evaluates DFEPN on two benchmark intent detection datasets:

  • SNIPS Natural Language Understanding benchmark (SNIPS): A widely used dataset for intent classification.
  • NLU-Evaluation Dataset (NLUE): Another standard benchmark for natural language understanding tasks.

The authors follow established data partitioning methods for these datasets to create scenarios for both FSID and GFSID. For GFSID, seen intents are randomly divided into auxiliary and base intents, while novel intents are also defined. The experiments utilize both "episodic" and "non-episodic" evaluation procedures to assess model performance under different testing conditions.

Key Insights and Ideas

  • Leveraging "Diversity Features": The central idea is that the differences between samples of the same intent (e.g., "King" - "Queen" analogy from word embeddings) can serve as valuable "diversity features." Transferring these features from well-represented intents to novel ones helps overcome the lack of expressive variety in the limited data for new intents.
  • Enhancing Novelty with Familiarity: By using abundant data from "seen" intents as a source for diversity features, the model can better represent the nuances and variations of "novel" intents, even with very few examples.
  • Importance of Semantic Encoder: The ablation study highlights that a strong semantic encoder (like BERT) is crucial. Without it, the diversity features can introduce noise, degrading performance.
  • Fusion is Key: The Feature Fusion module plays a significant role in effectively combining the original and diversity-enhanced features, showing substantial performance gains, especially in the challenging 1-shot scenario.
  • Query Enhancement Matters: Enhancing query vectors alongside support vectors helps the model adapt better to realistic scenarios and improves performance.

Key Results

DFEPN demonstrates superior performance compared to existing state-of-the-art methods on both SNIPS and NLUE datasets for both FSID and GFSID tasks.

  • Overall Improvement: Tables 2 and 3 show that DFEPN consistently achieves higher accuracy (h-acc, S-J, and S-N) across various settings (1-shot, 5-shot, episodic, non-episodic) compared to baselines like MN, PN, RN, HATT, MLMAN, HAPN, and SMAN.
  • Strong Performance in Challenging Scenarios: DFEPN shows significant improvements, particularly in the 1-shot setting and non-episodic evaluation, which are more challenging due to limited data and less constrained testing. For instance, on SNIPS, the full model achieves an h-acc of 97.06% in the 5-shot episodic setting.
  • Ablation Study Findings: The ablation studies confirm the effectiveness of each component. Removing the Diversity Feature Generator (DFG) or Feature Fusion (FF) module leads to noticeable performance drops, underscoring their importance.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that DFEPN performs slightly worse than SMAN in one specific scenario (10-way 1-shot S-J on NLUE with episodic evaluation). This is attributed to the testing uncertainty in smaller datasets where episodic evaluation might lead to repeated choices.
  • The performance decrease in non-episodic evaluation compared to episodic evaluation for most models (including DFEPN) indicates that non-episodic evaluation is inherently more challenging.
  • Future Work: The authors express their intention to explore the effectiveness of DFEPN in other tasks beyond intent detection.

Summary updated

BibTeX
@inproceedings{yang2022diversity,
  author = {Fengyi Yang and Xi Zhou and Yi Wang and Abibulla Atawulla and Ran Bi},
  title = {{Diversity Features Enhanced Prototypical Network for Few-shot Intent Detection}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/617},
}
Figure