Novel Intent Detection and Active Learning Based Classification (Student Abstract)

Ankan MullickAAAI 2023 · DOI

TL;DR

This paper introduces NIDAL, a novel semi-supervised framework that efficiently detects new user intents in conversational AI while minimizing human annotation costs. By combining a robust out-of-domain detection mechanism with an active learning strategy for handling rejected utterances, NIDAL significantly improves accuracy and reduces the annotation burden compared to existing methods.

What Problem Does This Paper Address?

Conversational AI systems, like chatbots and virtual assistants, need to adapt to new user intents as they emerge over time. This is crucial for maintaining user satisfaction and ensuring the system's continued relevance. However, retraining these systems to recognize novel intents presents a significant challenge: the sheer volume of new data makes manual labeling infeasible, and over-reliance on existing intents can degrade performance on those as well. Current approaches often struggle with a universal, end-to-end framework that can handle multiple novel intents across different languages and effectively reduce the human annotation effort required for misclassified or "rejected" utterances (those with low confidence scores).

Methods

NIDAL is an end-to-end, semi-supervised framework with two core components:

  • Novel Intent Detection (NID): This module identifies out-of-domain (OOD) samples from unlabeled data. It starts by training a neural model (JointBERT, based on BERT) on initial labeled data. Then, it uses the Hendrycks and Gimpel's Out-of-Domain Sample Detection (OOD-SD) algorithm to flag potential novel intents. A small percentage of these flagged samples are then manually annotated.
  • Active Learning (AL) for Rejected Utterances: This component addresses utterances that are classified as known intents but with low confidence scores. These "rejected" utterances are first passed to an XGBoost classifier for auto-correction. If still rejected by XGBoost, they are manually annotated. Both auto-corrected and manually annotated samples are added back to the training set, and the neural model is retrained. This process is iterated over several cycles to progressively improve accuracy.

The framework is designed to be language-agnostic.

Datasets

NIDAL was evaluated on several standard public datasets in the Natural Language Understanding (NLU) domain:

  • SNIPS: A dataset for intent classification in voice interfaces.
  • ATIS: A dataset for air travel information systems.
  • Facebook Multi-lingual (FB-EN, FB-ES, FB-TH): Datasets covering English, Spanish, and Thai languages, respectively, for intent classification.

Key Insights and Ideas

  • Unified Framework for Novelty and Rejection: The paper successfully integrates two critical aspects of conversational AI adaptation: detecting entirely new intents and refining the classification of existing ones when confidence is low.
  • Cost-Effective Annotation: By prioritizing manual annotation for a small subset of detected novel intents and using active learning for rejected utterances, NIDAL drastically reduces the human effort required.
  • Language Agnosticism: The use of BERT-based models, particularly the multilingual version, allows NIDAL to perform well across different languages without requiring language-specific model development.
  • Active Learning for Rejected Utterances is Crucial: The active learning component specifically targeting low-confidence known intents is highlighted as a key differentiator, addressing a gap in prior work that focused more on OOD detection.

Key Results

NIDAL demonstrates significant performance gains:

  • Accuracy and Macro-F1: The system outperforms baseline methods (SEEN, Zero-Shot-OOD, SENC-MaS) by more than 10% in terms of accuracy and macro-F1 score across various benchmark datasets. For example, on the FB-EN dataset, NIDAL achieved 98.0% accuracy and 93.1% macro-F1, significantly higher than the baselines.
  • Annotation Cost: NIDAL achieves these results while maintaining an annotation cost of only 6-10% of the available unlabeled data.

Limitations and Future Work

  • Limitations: The paper notes that competing baselines can only detect one novel class at a time, and for comparison, multiple novel intents in the test set were treated as a single novel class for these baselines.
  • Future Work: The authors suggest extending this work to more diverse domains, such as finance, technology, and healthcare, to further validate its generalizability.

Summary updated

BibTeX
@inproceedings{mullick2023novel,
  author = {Ankan Mullick},
  title = {{Novel Intent Detection and Active Learning Based Classification (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27003},
}
Figure