Fine-Tuning BERT for Schema-Guided Zero-Shot Dialogue State Tracking

Yu-Ping Ruan, Zhen-Hua Ling, Jia-Chen Gu, Quan LiuarXiv 2020 · OpenReview

Abstract

We present our work on Track 4 in the Dialogue System Technology Challenges 8 (DSTC8). The DSTC8-Track 4 aims to perform dialogue state tracking (DST) under the zero-shot settings, in which the model needs to generalize on unseen service APIs given a schema definition of these target APIs. Serving as the core for many virtual assistants such as Siri, Alexa, and Google Assistant, the DST keeps track of the user's goal and what happened in the dialogue history, mainly including intent prediction, slot filling, and user state tracking, which tests models' ability of natural language understanding. Recently, the pretrained language models have achieved state-of-the-art results and shown impressive generalization ability on various NLP tasks, which provide a promising way to perform zero-shot learning for language understanding. Based on this, we propose a schema-guided paradigm for zero-shot dialogue state tracking (SGP-DST) by fine-tuning BERT, one of the most popular pretrained language models. The SGP-DST system contains four modules for intent prediction, slot prediction, slot transfer prediction, and user state summarizing respectively. According to the official evaluation results, our SGP-DST (team12) ranked 3rd on the joint goal accuracy and 1st on the requested slots F1 among 25 participant teams.

Introduction

Dialogue State Tracking (DST) is a critical component of task-oriented dialogue systems. It is responsible for maintaining a representation of the user's goals and requests at each turn of a conversation. Traditionally, DST models are trained for a fixed set of services and slots (an ontology). However, this approach fails to scale to real-world applications where new services and APIs are introduced daily. Retraining massive models for every new integration is impractical.

A sample user utterance and the corresponding slots to be extracted by the DST system.

To address this, the research community has shifted toward zero-shot DST. In this setting, a model must generalize to unseen services by leveraging natural language descriptions of the service's API, known as a schema. This allows a virtual assistant to understand a user's intent for a service it has never encountered during training.

The SGP-DST Framework

We propose the Schema-Guided Paradigm for DST (SGP-DST), which leverages the pre-trained knowledge of BERT to interpret these schemas. The core idea is that if an API is described in plain English, a language model can use its existing linguistic understanding to map user utterances to new service parameters.

An example of a service schema definition, providing natural language descriptions for slots and intents.

The SGP-DST architecture is divided into four specialized modules:

  1. Intent Prediction: Identifies the user's high-level goal (e.g., CheckBalance or ReserveTable) by concatenating the dialogue history with intent descriptions from the schema.
  2. Slot Prediction: Extracts specific values from the conversation. This module distinguishes between "categorical" slots (fixed options like "checking" vs. "savings") and "free-form" slots (variable text like names or dates).
  3. Slot Transfer Prediction: Determines if a value should be carried over from a previous turn or a different domain. This is essential for multi-domain dialogues where a user might use a location mentioned in a weather search to book a nearby restaurant.
  4. User State Summarization: A logic layer that aggregates the outputs of the previous modules to update the final dialogue state.

Experimental Results

We evaluated SGP-DST on the Schema-Guided Dialogue (SGD) dataset, the largest public task-oriented dialogue corpus, featuring over 16,000 dialogues across 16 domains. The test set specifically includes unseen services to evaluate zero-shot performance.

Our system demonstrated strong generalization capabilities, ranking 1st in Requested Slots F1 and 3rd in Joint Goal Accuracy among 25 teams in the DSTC8 challenge. On the test set, the model achieved a Joint Goal Accuracy of 73.75%. While a performance gap remains between seen APIs (~88%) and unseen APIs (~67%), these results confirm that schema-guided zero-shot tracking is a viable path for scalable virtual assistants.

Key Insights and Future Work

Our analysis revealed that Slot Transfer is vital; without it, Joint Goal Accuracy on the development set dropped from approximately 80% to 40%. This highlights the importance of context management in complex, multi-turn interactions.

Future work will focus on mitigating error propagation inherent in pipeline architectures and improving the extraction of messy, free-form values. Additionally, exploring few-shot learning—where a tiny amount of target-domain data is provided—could bridge the remaining performance gap for production-ready systems.

Summary updated

Abstract

We present our work on Track 4 in the Dialogue System Technology Challenges 8 (DSTC8). The DSTC8-Track 4 aims to perform dialogue state tracking (DST) under the zero-shot settings, in which the model needs to generalize on unseen service APIs given a schema definition of these target APIs. Serving as the core for many virtual assistants such as Siri, Alexa, and Google Assistant, the DST keeps track of the user's goal and what happened in the dialogue history, mainly including intent prediction, slot filling, and user state tracking, which tests models' ability of natural language understanding. Recently, the pretrained language models have achieved state-of-the-art results and shown impressive generalization ability on various NLP tasks, which provide a promising way to perform zero-shot learning for language understanding. Based on this, we propose a schema-guided paradigm for zero-shot dialogue state tracking (SGP-DST) by fine-tuning BERT, one of the most popular pretrained language models. The SGP-DST system contains four modules for intent prediction, slot prediction, slot transfer prediction, and user state summarizing respectively. According to the official evaluation results, our SGP-DST (team12) ranked 3rd on the joint goal accuracy (primary evaluation metric for ranking submissions) and 1st on the requsted slots F1 among 25 participant teams.
BibTeX
@misc{ruan2020fine,
  author = {Yu-Ping Ruan and Zhen-Hua Ling and Jia-Chen Gu and Quan Liu},
  title = {{Fine-Tuning BERT for Schema-Guided Zero-Shot Dialogue State Tracking}},
  year = {2020},
  eprint = {2002.00181},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.00181},
}
Figure