Joint Multiple Intent Detection and Slot Filling via Self-distillation

Lisong Chen, Peilin Zhou, Yuexian Zou

Abstract

Intent detection and slot filling are two main tasks in natural language understanding (NLU) for identifying users' needs from their utterances. These two tasks are highly related and often trained jointly. However, most previous works assume that each utterance only corresponds to one intent, ignoring the fact that a user utterance in many cases could include multiple intents. In this paper, we propose a novel Self-Distillation Joint NLU model (SDJN) for multi-intent NLU. First, we formulate multiple intent detection as a weakly supervised problem and approach with multiple instance learning (MIL). Then, we design an auxiliary loop via self-distillation with three orderly arranged decoders: Initial Slot Decoder, MIL Intent Decoder, and Final Slot Decoder. The output of each decoder will serve as auxiliary information for the next decoder. With the auxiliary knowledge provided by the MIL Intent Decoder, we set Final Slot Decoder as the teacher model that imparts knowledge back to Initial Slot Decoder to complete the loop. The auxiliary loop enables intents and slots to guide mutually in-depth and further boost the overall NLU performance. Experimental results on two public multi-intent datasets indicate that our model achieves strong performance compared to others.

Introduction

In Natural Language Understanding (NLU), models typically perform two tasks: Intent Detection (identifying what the user wants) and Slot Filling (extracting specific entities like dates or locations). Traditionally, researchers assumed one utterance equals one intent. However, real-world users are often complex, packing multiple requests into a single sentence, such as "Book a table and play music."

Existing multi-intent models face two primary hurdles. First, they often use a single "context vector" for the whole sentence, missing fine-grained clues that distinguish similar intents. Second, most models use intent to help find slots, but rarely use slot information to help identify the intent. The authors argue that these two tasks are deeply interrelated and should "talk" to each other in a loop rather than a straight line.

Methodology

The authors propose the Self-Distillation Joint NLU (SDJN) model, which consists of a shared encoder and a unique three-decoder "auxiliary loop."

The SDJN architecture featuring the auxiliary loop and the MIL Intent Decoder.

The loop consists of three orderly arranged decoders:

  1. Initial Slot Decoder: A simple decoder that makes a first pass at identifying slots.
  2. MIL Intent Decoder: This treats the utterance as a "bag of words" using Multiple Instance Learning (MIL). It uses the output from the Initial Slot Decoder to help predict intent distributions at the token level, aggregating them to find the overall intents.
  3. Final Slot Decoder: This acts as the "Teacher." It uses a Graph Attention Network (GAT) to explicitly model the relationships between the predicted intents and specific tokens to produce the final, high-quality slot labels.

To close the loop, the authors use Self-Distillation. The Final Slot Decoder (which has the most information) provides "hints" (via Mean Squared Error loss on hidden states) to the Initial Slot Decoder. This forces the earlier stage of the model to learn better intermediate representations, creating a virtuous cycle where better slots lead to better intents, which lead to even better slots.

Experimental Setup

The model was evaluated on two public multi-intent NLU benchmarks:

  • MixATIS: A multi-intent variant of the Airline Travel Information System dataset.
  • MixSNIPS: A multi-intent version of the SNIPS personal assistant dataset. Both datasets contain utterances with 1 to 3 intents, with a distribution of roughly 30% single-intent, 50% dual-intent, and 20% triple-intent sentences.

Results and Analysis

The SDJN model outperformed several state-of-the-art baselines, including AGIF and Stack-Propagation, across both datasets.

Performance comparison of SDJN against state-of-the-art models on MixATIS and MixSNIPS datasets.

Key findings include:

  • Overall Accuracy: On MixATIS, SDJN achieved 44.6%, a significant jump over the previous best of 40.8%. On MixSNIPS, it reached 75.7% (up from 74.2%).
  • Mutual Guidance: The paper demonstrates that knowing a "restaurant_type" slot exists is a massive hint that the intent is "BookRestaurant," which in turn helps refine other slots.
  • Hints over Soft Targets: In their distillation process, the authors found that providing "hints" (internal hidden states) from the teacher model was more effective than "soft targets" (final probability outputs). This suggests that how the teacher thinks is more important for the student than just what the teacher concludes.

Conclusion

SDJN effectively addresses the multi-intent NLU challenge by acknowledging that intents are localized and that slots and intents should guide each other mutually. While the model shows strong performance, future work could explore reducing computational complexity and implementing adaptive intent thresholds to handle highly ambiguous cases.

Summary updated

Abstract

Intent detection and slot filling are two main tasks in natural language understanding (NLU) for identifying users' needs from their utterances. These two tasks are highly related and often trained jointly. However, most previous works assume that each utterance only corresponds to one intent, ignoring the fact that a user utterance in many cases could include multiple intents. In this paper, we propose a novel Self-Distillation Joint NLU model (SDJN) for multi-intent NLU. First, we formulate multiple intent detection as a weakly supervised problem and approach with multiple instance learning (MIL). Then, we design an auxiliary loop via self-distillation with three orderly arranged decoders: Initial Slot Decoder, MIL Intent Decoder, and Final Slot Decoder. The output of each decoder will serve as auxiliary information for the next decoder. With the auxiliary knowledge provided by the MIL Intent Decoder, we set Final Slot Decoder as the teacher model that imparts knowledge back to Initial Slot Decoder to complete the loop. The auxiliary loop enables intents and slots to guide mutually in-depth and further boost the overall NLU performance. Experimental results on two public multi-intent datasets indicate that our model achieves strong performance compared to others.
BibTeX
@inproceedings{chen2022joint,
  author = {Lisong Chen and Peilin Zhou and Yuexian Zou},
  title = {{Joint Multiple Intent Detection and Slot Filling via Self-distillation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=BvbZ3s_CDmJ},
}
Figure