SGD-QA: Fast Schema-Guided Dialogue State Tracking for Unseen Services

Yang Zhang, Vahid Noroozi, Evelina Bakhturina, Boris GinsburgarXiv 2021 · OpenReview

Abstract

Dialogue state tracking is an essential part of goal-oriented dialogue systems, while most of these state tracking models often fail to handle unseen services. In this paper, we propose SGD-QA, a simple and extensible model for schema-guided dialogue state tracking based on a question answering approach. The proposed multi-pass model shares a single encoder between the domain information and dialogue utterance. The domain's description represents the query and the dialogue utterance serves as the context. The model improves performance on unseen services by at least 1.6x compared to single-pass baseline models on the SGD dataset. SGD-QA shows competitive performance compared to state-of-the-art multi-pass models while being significantly more efficient in terms of memory consumption and training performance.

Introduction

Building virtual assistants that scale across diverse services is a significant challenge in Dialogue State Tracking (DST). Traditional models are often "ontology-specific," relying on a fixed set of slots and values. This approach fails when encountering "unseen" services—new domains or APIs not present during training. To address this, researchers utilize Schema-Guided datasets where models are provided with natural language descriptions of services. While multi-pass models generalize well to these new schemas, they are often computationally expensive and slow. SGD-QA addresses these issues by framing DST as a streamlined Question Answering task.

Methodology: DST as Question Answering

The core innovation of SGD-QA is framing DST as a Question Answering (QA) task. Instead of predicting slots from a fixed list, the model treats the schema description (e.g., "the number of people for the reservation") as a query and the dialogue turn as the context.

Single Shared Encoder Architecture

Unlike previous multi-pass architectures (like SPPD) that use separate encoders for different components, SGD-QA employs a single shared BERT encoder. This design allows the model to learn a unified representation for intents, requested slots, and slot values, significantly reducing the parameter footprint.

Comparison of model architectures: (a) SGD baseline, (b) SPPD, and (c) the proposed SGD-QA which utilizes a single shared BERT encoder.

The architecture includes specialized heads:

  • Classification heads for binary (intent activation) and 3-way tasks (slot status: active, none, or "dontcare").
  • Span prediction head for extracting specific text values from the dialogue, similar to SQuAD-style extraction.

To optimize training, the authors used data balancing to manage the high ratio of negative examples and automatic mixed precision for faster computation. This allows the model to converge in just 3 epochs, a massive improvement over the 80–160 epochs required by single-pass baselines.

Evaluation and Results

The model was evaluated on the Schema-Guided Dialogue (SGD) dataset, the largest public DST dataset with over 20,000 dialogues across 20 domains. The evaluation focuses on zero-shot generalization to unseen services.

Performance and Efficiency

SGD-QA demonstrates a significant performance leap, outperforming single-pass baselines by at least 1.6x on unseen services. In single-domain tasks, it achieved a Joint Goal Accuracy (Joint GA) of 65%, a 1.3x improvement over the standard SGD baseline.

Performance comparison of SGD-QA against state-of-the-art models on the SGD dataset, highlighting improvements in Joint Goal Accuracy.

Beyond accuracy, SGD-QA is remarkably efficient. By consolidating tasks into a single encoder, it reduces memory consumption to approximately 1/5th or 1/6th of previous multi-pass models like SPPD. Furthermore, because the queries are independent, they can be parallelized during inference to maintain low latency.

Discussion and Conclusion

The success of SGD-QA highlights the power of framing DST as a reading comprehension task. By focusing on the description of a slot rather than its identity, the model generalizes effectively to new domains. While challenges remain—such as the inherent difficulty of extractive tasks compared to classification—SGD-QA sets a new standard for efficient, scalable, and generalizable dialogue state tracking.

Summary updated

Abstract

Dialogue state tracking is an essential part of goal-oriented dialogue systems, while most of these state tracking models often fail to handle unseen services. In this paper, we propose SGD-QA, a simple and extensible model for schema-guided dialogue state tracking based on a question answering approach. The proposed multi-pass model shares a single encoder between the domain information and dialogue utterance. The domain's description represents the query and the dialogue utterance serves as the context. The model improves performance on unseen services by at least 1.6x compared to single-pass baseline models on the SGD dataset. SGD-QA shows competitive performance compared to state-of-the-art multi-pass models while being significantly more efficient in terms of memory consumption and training performance. We provide a thorough discussion on the model with ablation study and error analysis.
BibTeX
@misc{zhang2021sgd,
  author = {Yang Zhang and Vahid Noroozi and Evelina Bakhturina and Boris Ginsburg},
  title = {{SGD-QA: Fast Schema-Guided Dialogue State Tracking for Unseen Services}},
  year = {2021},
  eprint = {2105.08049},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.08049},
}
Figure