QA-Driven Zero-shot Slot Filling with Weak Supervision Pretraining

Xinya Du, Luheng He, Qi Li, Dian Yu, Panupong Pasupat, Yuan Zhang

Introduction

In task-oriented dialog systems, slot filling is the critical process of extracting semantic entities from user utterances. For instance, in the request "Find me a flight to Paris," a system must identify "Paris" as the destination_city. The primary challenge lies in Zero-shot Slot Filling (ZSSF): the ability of a model to generalize to entirely new domains (e.g., from "Travel" to "Medical") without additional training data.

Traditional approaches rely on sequence tagging (BIO tagging) based on cryptic slot descriptions, which often fails to capture the semantic nuances required for cross-domain transfer. This paper introduces a novel framework that reformulates this task as a Question Answering (QA) problem.

The QASF Framework

The authors propose QASF (QA-driven Slot Filling), a paradigm shift that moves away from token-level tagging toward reading comprehension. Instead of tagging tokens, the model is asked to find the answer to a natural language question within the user's utterance.

Comparison between traditional BIO tagging and the proposed QA-driven slot filling approach.

Question Generation Strategy

A core component of QASF is the transformation of ambiguous slot names into descriptive questions. Using a rule-based system, raw descriptions like "artist" are converted into queries such as "Who is the artist to play?" This allows the model to leverage the rich semantic priors of large language models (like BERT), which already understand that "Who" typically refers to persons or entities.

Examples of converting raw slot descriptions into natural language questions.

Span-based QA Model

The architecture utilizes a BERT-based model that takes a concatenated input of the generated question and the user utterance. Rather than classifying each token into a BIO category, the model predicts the start and end indices of the relevant span. This formulation is inherently more robust for zero-shot scenarios as it focuses on the relationship between the question's intent and the utterance's content.

Weak Supervision Pretraining

To further enhance performance, the authors introduce a weak supervision stage. They utilize an ontology of known slot values and unlabeled conversations from MultiWOZ 2.2 to generate over 67,000 synthetic QA pairs. Pretraining on this "silver" data teaches the model the structural mechanics of the QA task before fine-tuning on domain-specific benchmarks like SNIPS and TOP.

Experimental Results

The QASF model demonstrates significant improvements over existing baselines:

  • State-of-the-Art Gains: QASF outperformed previous leading models like Coach by 7% on the SNIPS dataset.
  • Impact of Pretraining: The weak supervision stage provided a significant boost, improving F1 scores by an average of 2.9% and up to 20% in specific zero-shot settings on the TOP dataset.
  • Semantic Robustness: The QA approach proved more effective than standard tagging (ZSBT) at handling "semantic shifts," where the same slot name might refer to different entity types across domains (e.g., "object_type" referring to a book vs. a movie time).

Conclusion and Future Work

By reframing slot filling as a QA task, QASF demonstrates superior zero-shot generalization. While highly effective, the authors note limitations regarding dataset artifacts—where models might memorize common words—and the current reliance on rule-based question templates. Future research will explore more sophisticated, human-like question generation and methods to better align semantics across drastically different domains.

Summary updated

BibTeX
@inproceedings{du2021qa,
  author = {Xinya Du and Luheng He and Qi Li and Dian Yu and Panupong Pasupat and Yuan Zhang},
  title = {{QA-Driven Zero-shot Slot Filling with Weak Supervision Pretraining}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.83},
}
Figure