A Domain-Transfer Meta Task Design Paradigm for Few-Shot Slot Tagging

Fengyi Yang, Xi Zhou, Yating Yang, Bo Ma, Rui Dong, Abibulla AtawullaAAAI 2023 · OpenReview

TL;DR

This paper introduces a novel meta-task design for few-shot slot tagging that addresses the issue of overlapping slots between different domains, a common problem in real-world dialogue systems. Their approach leverages a "basic domain" with abundant data to transfer historical knowledge, significantly improving performance on limited target domain data and achieving state-of-the-art results.

What Problem Does This Paper Address?

Slot tagging, a crucial component of dialogue systems, aims to extract specific information (slots) from user utterances. Few-shot slot tagging is particularly important for new dialogue systems where obtaining large amounts of labeled data is impractical. Traditional few-shot learning methods often rely on meta-learning, where models are trained on numerous "meta-tasks" designed to mimic the low-data scenario. However, these methods often overlook a critical real-world phenomenon: domains frequently share common slots (e.g., "time," "number of people"). When meta-tasks treat these overlapping slots as distinct across domains, it can lead to "historical information forgetting," hindering the model's ability to leverage past knowledge effectively. This paper tackles this by proposing a meta-task design that explicitly accounts for and utilizes these overlapping slots.

Methods

The core innovation lies in their domain-transfer meta task design paradigm. Instead of randomly sampling meta-tasks, they first identify a "basic domain" for each target domain based on the degree of slot label overlap (measured by Jaccard similarity). This basic domain, which has abundant data, serves as a source of historical knowledge.

Their proposed model, the Linguistic Features Enhanced Task Adaptation Network (LFETAN), operates in two phases:

  1. Basic Slot Tagging Network (BSTN): This phase trains a model on the auxiliary set (large data from the basic domain) using a supervised approach with a Conditional Random Field (CRF) framework. This captures historical information related to overlapping slots.
  2. Task Adaptation Meta Network (TAMN): This phase trains a meta-model on the target domain's limited data (support set) and the auxiliary set. It aims to learn meta-knowledge for novel slots while fusing it with the historical information from the BSTN. A Task Adaptation module is key here, merging historical and meta-knowledge.

Furthermore, LFETAN incorporates Part-of-Speech (POS) features to enhance word embeddings, acting as a universal cross-domain knowledge source. A POS Encoder is trained to capture these features, which are then fused with other representations. The overall loss function combines meta-learning, POS adaptation, and task adaptation losses.

Datasets

The authors evaluate their method on two benchmark datasets:

  • SNIPS: A well-known slot tagging dataset with 7 domains.
  • NER (Named Entity Recognition): Used to demonstrate the generalizability of their approach to a related sequence labeling task, treating different NER datasets as distinct domains.

They follow standard data splits and domain assignments from previous work (e.g., Hou et al., 2020) for fair comparison.

Key Insights and Ideas

  • Overlap is Key: The paper highlights that ignoring overlapping slots in meta-task design is a significant flaw that leads to performance degradation in few-shot scenarios.
  • Basic Domain Strategy: Assigning a "basic domain" based on slot similarity is an effective way to leverage abundant historical data for domains with limited resources.
  • Two-Phase Learning: Separating the learning of historical (overlap) knowledge from meta-knowledge (novel slots) in distinct phases allows for more targeted knowledge transfer.
  • POS Features Matter: Incorporating POS tags as linguistic features provides a valuable, generalizable signal that aids in learning representations, especially in low-resource settings.
  • Model Stability: Their approach shows more stable performance across different domains compared to some existing methods, particularly in challenging few-shot scenarios.

Key Results

The proposed LFETAN model achieves state-of-the-art performance on both the SNIPS and NER datasets for few-shot slot tagging.

  • On SNIPS, their model consistently outperforms baselines in both 1-shot and 5-shot settings, with average F1 scores being 2.46 and 1.32 higher than the previous best (MCML) for 1-shot and 5-shot, respectively.
  • On NER, they also show significant improvements, outperforming InversePrompt by F1 scores of 16.47 and 19.71 on average for 1-shot and 5-shot, respectively.
  • Ablation studies confirm the effectiveness of each component: the novel meta-task design (BSTN) has the most significant impact, especially in the 1-shot setting, followed by the Task Adaptation Network (TAMN) and POS features.

Limitations and Future Work

  • Domain Specificity: The paper notes that the "Cr" domain in SNIPS has a unique characteristic (few slots, all overlapping) which benefits MCML's specialized module. While their model is more stable, this suggests that highly specialized domain structures might require further tailored approaches.
  • Future Work: The authors suggest exploring the effectiveness of their domain-transfer meta-task design paradigm in other related tasks beyond slot tagging and NER.

Summary updated

Abstract

Few-shot slot tagging is an important task in dialogue systems and attracts much attention of researchers. Most previous few-shot slot tagging methods utilize meta-learning procedure for training and strive to construct a large number of different meta tasks to simulate the testing situation of insufficient data. However, there is a widespread phenomenon of overlap slot between two domains in slot tagging. Traditional meta tasks ignore this special phenomenon and cannot simulate such realistic few-shot slot tagging scenarios. It violates the basic principle of meta-learning which the meta task is consistent with the real testing task, leading to historical information forgetting problem. In this paper, we introduce a novel domain-transfer meta task design paradigm to tackle this problem. We distribute a basic domain to each target domain based on the coincidence degree of slot labels between these two domains. Unlike classic meta tasks which only rely on small samples of target domain, our meta tasks aim to correctly infer the class of target domain query samples based on both abundant data in basic domain and scarce data in target domain. To accomplish our meta task, we propose a Task Adaptation Network to effectively transfer the historical information from the basic domain to the target domain. We carry out sufficient experiments on the benchmark slot tagging dataset SNIPS and the name entity recognition dataset NER. Results demonstrate that our proposed model outperforms previous methods and achieves the state-of-the-art performance.
BibTeX
@inproceedings{yang2023domain,
  author = {Fengyi Yang and Xi Zhou and Yating Yang and Bo Ma and Rui Dong and Abibulla Atawulla},
  title = {{A Domain-Transfer Meta Task Design Paradigm for Few-Shot Slot Tagging}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=vHXF7Z2Ipm},
}
Figure