Enriched Pre-trained Transformers for Joint Slot Filling and Intent Detection

Momchil Hardalov, Ivan Koychev, Preslav Nakov

Abstract

Detecting the user's intent and finding the corresponding slots among the utterance's words are important tasks in natural language understanding (NLU). Their interconnected nature makes their joint modeling a standard part of training such models. Moreover, data scarceness and specialized vocabularies pose additional challenges. Recently, the advances in pre-trained language models, namely contextualized models such as ELMo and BERT, have revolutionized the field by tapping the potential of training very large models with just a few steps of fine-tuning on a task-specific dataset. Here, we leverage such models, namely BERT and RoBERTa, and we design a novel architecture on top of them. Moreover, we propose an intent pooling attention mechanism, and we reinforce the slot filling task by fusing intent distributions, word features, and token representations. The experimental results on standard datasets show that our model outperforms both the current non-BERT state of the art as well as some stronger BERT-based baselines.

Introduction

In Natural Language Understanding (NLU), virtual assistants must perform two primary tasks: Intent Detection (classifying the user's goal, e.g., "PlayMusic") and Slot Filling (identifying specific entities, e.g., "Artist: Justin Broadrick"). While these tasks are highly interdependent, many existing models treat them with insufficient cross-talk or rely on the [CLS] token alone to represent the entire sentence's intent. Furthermore, NLU often struggles with "out-of-vocabulary" words and specialized entities.

This paper introduces Transformer-NLU, an enhanced architecture that builds on pre-trained models like BERT and RoBERTa to jointly solve intent detection and slot filling. The authors aim to bridge the gap between these two tasks by allowing the predicted intent to explicitly guide the slot-filling process while leveraging the massive "world knowledge" already embedded in pre-trained Transformers.

The Transformer-NLU Architecture

Comparison of the baseline BERT-Joint architecture (a) and the proposed Transformer-NLU architecture (b).

The proposed Transformer-NLU architecture introduces three key enhancements over standard BERT-based joint models:

1. Intent Pooling Attention

Instead of relying solely on the [CLS] token for intent classification, the model learns an alignment function to calculate attention weights for every token in the sequence. This creates a "pooled" representation of the entire utterance, ensuring the intent classification is informed by the most relevant words, such as key verbs or nouns.

Visualization of the intent pooling attention mechanism for an ATIS flight query, showing how specific words contribute to the intent representation.

2. Feature Fusion for Slots

The slot-filling layer is "enriched" by concatenating three distinct information sources: the hidden representation from the Transformer, the predicted intent distribution, and additional word features. This explicit feedback loop allows the model's confidence in an intent (e.g., PlayMusic) to guide the identification of ambiguous slots (e.g., labeling a string as an Artist).

3. Linguistic Pre-processing

To handle the messy nature of user queries, the model uses a "TrueCase" model to fix capitalization and an external Named Entity Recognition (NER) tool. These features are passed through a two-layer feed-forward network before being fused into the slot-filling task.

Experimental Evaluation

The model was evaluated on two widely used NLU benchmarks:

  • ATIS (Airline Travel Information System): A smaller, domain-specific dataset focused on flight reservations, featuring 120 slot categories and ~4,500 training samples.
  • SNIPS: A larger, more diverse dataset gathered from personal voice assistants, covering seven intents (e.g., weather, restaurant reservations) and ~13,000 training samples.

Results and Analysis

The Transformer-NLU:BERT model set new performance standards across both datasets:

  • Relative Error Reduction (RER): Compared to standard BERT-Joint baselines, the model reduced errors by 17.4% (Intent) and 19.8% (Slots) on ATIS. On SNIPS, error reductions exceeded 55% across several metrics.
  • Accuracy: On the SNIPS dataset, the model achieved an intent accuracy of 98.86% and a slot F1-score of 96.57%.
  • Sentence-level Accuracy: Requiring both the intent and all slots to be correct, the model reached 92.14% on SNIPS, a significant jump over previous non-Transformer state-of-the-art models (~80-86%).

Key Insights

  • BERT as a Knowledge Base: Knowledge analysis revealed that 66-69% of slots in these datasets match Wikipedia titles. By using BERT, the model leverages implicit knowledge learned during pre-training to generalize to entities not seen in the NLU training set.
  • The CRF Paradox: Interestingly, adding a Conditional Random Field (CRF) layer—a standard tool for sequence tagging—actually degraded performance. This is likely due to large gradients from the Transformer making CRF optimization unstable on smaller datasets.
  • Backbone Sensitivity: While RoBERTa is generally considered a stronger model, it performed slightly worse than BERT in this context, possibly due to the difference in pre-training data (CommonCrawl vs. Wikipedia).

Conclusion

Transformer-NLU demonstrates that enriching pre-trained Transformers with intent pooling attention and feature fusion significantly improves joint NLU task performance. Future work will explore hierarchical slots and more advanced NER frameworks to further refine the feature fusion process.

Summary updated

Abstract

Detecting the user's intent and finding the corresponding slots among the utterance's words are important tasks in natural language understanding. Their interconnected nature makes their joint modeling a standard part of training such models. Moreover, data scarceness and specialized vocabularies pose additional challenges. Recently, the advances in pre-trained language models, namely contextualized models such as ELMo and BERT have revolutionized the field by tapping the potential of training very large models with just a few steps of fine-tuning on a task-specific dataset. Here, we leverage such models, namely BERT and RoBERTa, and we design a novel architecture on top of them. Moreover, we propose an intent pooling attention mechanism, and we reinforce the slot filling task by fusing intent distributions, word features, and token representations. The experimental results on standard datasets show that our model outperforms both the current non-BERT state of the art as well as some stronger BERT-based baselines.
BibTeX
@inproceedings{hardalov2023enriched,
  author = {Momchil Hardalov and Ivan Koychev and Preslav Nakov},
  title = {{Enriched Pre-trained Transformers for Joint Slot Filling and Intent Detection}},
  booktitle = {RANLP},
  year = {2023},
  url = {https://openreview.net/forum?id=-SQMbrJmKw},
}
Figure