A Multitask Active Learning Framework for Natural Language Understanding

Hua Zhu, Wu Ye, Sihan Luo, Xidong ZhangCOLING 2020

TL;DR

This paper introduces a novel multitask active learning framework that leverages the inherent relationships between intent detection and slot filling in Natural Language Understanding (NLU). By jointly modeling these tasks, the framework significantly reduces the amount of labeled data needed to achieve high performance, making NLU systems more efficient to train.

What Problem Does This Paper Address?

Training robust Natural Language Understanding (NLU) systems, which are crucial for voice assistants and chatbots, typically requires vast amounts of labeled data. This data annotation process is expensive and time-consuming. Active Learning (AL) offers a solution by intelligently selecting the most informative unlabeled samples for annotation. However, existing AL methods for NLU often treat intent detection (identifying the user's goal, e.g., "book a flight") and slot filling (extracting specific information, e.g., "from London to Paris") as separate tasks, neglecting the valuable interdependencies between them. This paper addresses the gap of effectively utilizing these task relationships within an active learning setting to improve data efficiency.

Methods

The core of this work is a multitask active learning framework designed for NLU. The framework employs a joint model that simultaneously predicts user intent and extracts semantic slots. This joint modeling approach inherently captures the relationships between these two sub-tasks.

The paper proposes using this joint model to enable pool-based AL algorithms to query samples that are most informative for both tasks. Specifically, they implement and adapt three common uncertainty sampling strategies:

  • Least Confidence (LC): Selects samples where the model is least confident about its prediction.
  • Margin Sampling (M): Chooses samples with the smallest margin between the top two most likely predictions.
  • Entropy (H): Identifies samples that have the highest uncertainty across all possible predictions.

A key technical contribution is an efficient computation for the joint entropy of the NLU model, which is crucial for the Entropy query strategy. This is achieved through a dynamic programming approach, building upon existing work on linear-chain CRFs. The framework is also shown to be effective when using BERT (Bidirectional Encoder Representations from Transformers) as the underlying NLU model.

Datasets

The experiments were conducted on several publicly available NLU datasets:

  • ATIS: A widely used dataset for spoken language understanding, focusing on flight-related information.
  • SNIPS: A dataset collected from a personal voice assistant, covering various user intents.
  • Facebook-Weather, Facebook-Alarm, Facebook-Reminder: Datasets from Schuster et al. (2019) covering three distinct domains.

The paper does not introduce new benchmarks but rather evaluates its framework on existing, well-established NLU datasets.

Key Insights and Ideas

  • Leveraging Task Relationships is Key: The most significant insight is that explicitly modeling and utilizing the relationship between intent detection and slot filling within an active learning framework leads to substantial improvements in data efficiency. This suggests that NLU tasks are not truly independent and benefit from joint consideration.
  • Joint Entropy is More Informative: The paper demonstrates that using the joint entropy (considering both intent and slots) as a query strategy is more effective than using separate "Intent Entropy" or "Slot Entropy." This highlights the importance of a holistic view of uncertainty for effective sample selection.
  • Framework Agnosticism: The proposed framework is shown to be effective even when the underlying NLU model is switched from a BiLSTM-based model to a powerful pre-trained model like BERT. This indicates the robustness and generalizability of the active learning approach.
  • Approximation vs. Exact Computation: While exact computation of entropy is more computationally intensive, it yields better performance than approximations, underscoring the value of precise uncertainty estimation.

Key Results

  • The proposed multitask active learning framework consistently achieves competitive performance with significantly less training data compared to baseline methods (including random sampling and a pipeline-based active learning approach).
  • The framework achieved over 40% reduction in the number of samples needed (Qneeded) compared to random sampling across various datasets.
  • The Entropy query strategy, utilizing the efficient joint entropy computation, generally performed better than other uncertainty sampling methods like Least Confidence and Margin Sampling.
  • When using BERT as the base model, the framework still demonstrated superior data efficiency over random sampling, even with the increased complexity of the base model.

Limitations and Future Work

  • Computational Cost of Exact Entropy: The paper acknowledges that the exact computation of joint entropy is more computationally expensive than approximations, which might be a consideration for real-time applications.
  • Future Work: The authors suggest that their framework has the potential for industrial application. Further research could explore more advanced query strategies or investigate the framework's performance on even more diverse and challenging NLU tasks. The paper also hints at the possibility of exploring other multitask learning paradigms within the active learning context.

Summary updated

BibTeX
@inproceedings{zhu2020multitask,
  author = {Hua Zhu and Wu Ye and Sihan Luo and Xidong Zhang},
  title = {{A Multitask Active Learning Framework for Natural Language Understanding}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
}
Figure