Transformer-Capsule Model for Intent Detection (Student Abstract)

Aleksander Obuchowski, Michal LewAAAI 2020 · DOI

Abstract

Intent detection is a critical component of modern Natural Language Understanding (NLU) systems, responsible for mapping user queries to specific semantic labels. While Transformer-based models have set new benchmarks in many NLP tasks, their ability to filter and route features can be further optimized. This paper introduces a hybrid Transformer-Capsule architecture that combines the contextual attention of Transformers with the robust feature-routing mechanisms of Capsule Networks. Our model achieves state-of-the-art results on the AskUbuntu, WebApp, and ATIS datasets, demonstrating superior performance particularly in low-resource scenarios.

Introduction

In the field of NLU, intent detection serves as the foundation for interactive agents and chatbots. Categorizing a query like "Book a flight to London" into a BookFlight intent requires a model that can capture both local word importance and global semantic structure. Traditional approaches using Recurrent Neural Networks (RNNs) often struggle with long-range dependencies. While Transformers have addressed these issues through self-attention, there remains a challenge in how captured features are aggregated and passed through the network to minimize noise and maximize representational richness.

Methodology

The proposed Transformer-Capsule architecture consists of three primary stages designed to refine raw text into a high-level intent representation.

1. Transformer Encoder

The pipeline begins with pre-trained GloVe embeddings, which are processed by a single-layer Transformer encoder. This module utilizes a 12-head self-attention mechanism to identify contextual relationships between words, producing a set of context-aware vectors.

2. Capsule Module

Instead of employing standard pooling or fully connected layers, the Transformer's output is passed into a Capsule Network. This module comprises 100 capsules, each with 15 dimensions.

  • Dynamic Routing: The model employs four iterations of dynamic routing. This iterative process allows the network to selectively pass information from lower-level capsules to higher-level ones based on feature relevance, effectively filtering out noise.
  • Squash Activation: To preserve the directional information of the capsules, a "squash" function is applied, normalizing the length of the vectors while maintaining their orientation.
Architecture of the Transformer-Capsule model showing the flow from input embeddings through the Transformer and Capsule modules to the final output class.

3. Classification

The final capsule outputs are flattened and fed into a dense layer with a softmax activation function to produce the final intent label.

Experimental Evaluation

The model was evaluated on three benchmark datasets: AskUbuntu (technical queries), WebApp (web application intents), and ATIS (airline travel information). We compared our hybrid approach against established NLU frameworks including RASA, Snips, and IBM Watson, as well as previous state-of-the-art methods like Subword Semantic Hashing (SSH) and WAIS.

Results

The Transformer-Capsule model consistently outperformed all baselines across the tested datasets.

Comparison of Micro F1 scores on the AskUbuntu and WebApp datasets, showing the Transformer-Capsule model outperforming several industry-standard NLU frameworks.

On the AskUbuntu dataset, the model achieved a Micro F1 score of 0.98, a significant improvement over the previous SOTA of 0.94. Similarly, on the WebApp dataset, it reached a score of 0.92, surpassing the 0.85 benchmark. For the larger ATIS dataset, the model achieved 98.89% accuracy, edging out WAIS (98.61%) and the original Capsule-NLU implementation (95.0%).

Discussion

The success of this architecture stems from the synergy between attention and routing. While the Transformer's attention mechanism excels at identifying which words in a sequence are relevant to one another, the Capsule Network's dynamic routing is superior at identifying which features or "entities" are most salient for the final classification.

Furthermore, the model demonstrates high efficiency; achieving state-of-the-art results with only a single Transformer layer suggests that the Capsule layer adds significant representational power. This reduces the need for the extremely deep stacks typically found in models like BERT, making it a potentially more lightweight alternative for specific NLU tasks.

Conclusion and Future Work

The Transformer-Capsule model represents a significant step forward in robust intent detection, particularly for smaller datasets where generalization is difficult. Future research will focus on extending this architecture to joint intent detection and slot-filling tasks. Additionally, the authors plan to investigate the impact of pre-training the Transformer encoder on larger corpora to further enhance performance.

Summary updated

BibTeX
@inproceedings{obuchowski2020transformer,
  author = {Aleksander Obuchowski and Michal Lew},
  title = {{Transformer-Capsule Model for Intent Detection (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7215},
}
Figure