Few-shot Learning for Slot Tagging with Attentive Relational Network

Cennet Oguz, Ngoc Thang Vu

Abstract

Slot tagging is a critical component of natural language understanding, yet it typically requires extensive labeled datasets. This paper proposes the Attentive Relational Network (ARN), a few-shot learning framework designed to recognize new information categories using only a few examples. By integrating contextual embeddings (BERT and ELMo) with a novel learnable attention mechanism, ARN identifies relevant features while suppressing noise. Experimental results on the SNIPS dataset demonstrate that ARN consistently outperforms standard metric-based approaches, providing a robust solution for low-resource slot tagging.

Introduction

In Natural Language Processing (NLP), slot tagging identifies specific entities within user queries, such as artist names or locations. While essential for virtual assistants, training these models is data-intensive. Adding a new domain, like "Medical Appointments," often presents a "low-resource" challenge where only a handful of labeled examples are available. Existing few-shot methods often rely on expensive external slot descriptions or inconsistent label-to-token relationships. This work introduces an approach that performs few-shot slot tagging using only small annotated samples, without the need for external linguistic metadata.

Methodology: Attentive Relational Networks

The proposed architecture builds upon metric-based learning, which learns a similarity function between samples. The Attentive Relational Network (ARN) introduces three primary enhancements:

  1. Contextual Embeddings: Unlike static vectors, ARN utilizes ELMo and BERT to capture word meanings based on surrounding context, which is crucial for distinguishing ambiguous slots.
  2. Learnable Attention Estimator: The core innovation is an attention module that uses convolutional blocks to produce local descriptors. This module learns to highlight relevant features and suppress misleading ones when comparing a "support" sample to a "query" input.
  3. Episode-Based Meta-Learning: The model is trained using episodes that mimic the few-shot task, where it must predict labels for a query set based on a small support set.
Comparison of metric-based architectures and the detailed architecture of the proposed Attentive Relational Module.

Experimental Setup

The model was evaluated using the SNIPS dataset, comprising 39 slots across 7 domains (e.g., Weather, Music, Restaurant Booking). To test generalization, a "leave-one-domain-out" strategy was employed: the model was trained on six domains and tested on the seventh, unseen domain. Performance was measured across different "shots" ($K=5, 10, 15$) and varying collection sizes to assess how data availability impacts accuracy.

Results and Discussion

The Attentive Relational Network consistently outperformed other metric-based models, such as Matching Networks and Prototypical Networks, across nearly all domains and shot counts.

Performance results of the Attentive Relational Network across different domains, shot counts, and embedding types.

Key findings include:

  • Contextual Advantage: Moving from static vectors (FastText) to contextual embeddings (BERT/ELMo) provided the most significant performance boost, proving that the "neighborhood" of a word is essential for identification.
  • Attention as a Filter: The attention mechanism effectively filters noise, allowing the model to focus on local features that signal specific slot types while ignoring misleading matches.
  • Embedding Specialization: ELMo showed superior performance on proper nouns (e.g., specific locations in FindScreeningEvent), while BERT excelled in slots containing common nouns.

Conclusion

The Attentive Relational Network represents a significant advancement in few-shot slot tagging. By combining the power of contextual embeddings with a learnable attention mechanism, the model achieves high accuracy in low-resource scenarios. Future work will explore deeper domain-specific integration and hybrid models to further address the challenges of unique proper nouns and complex linguistic structures.

Summary updated

BibTeX
@inproceedings{oguz2021few,
  author = {Cennet Oguz and Ngoc Thang Vu},
  title = {{Few-shot Learning for Slot Tagging with Attentive Relational Network}},
  booktitle = {EACL},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.134},
}
Figure