SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems

Harrison Lee, Raghav Gupta, Abhinav Rastogi, Yuan Cao, Bin Zhang, Yonghui Wu

Abstract

Modern task-oriented dialogue systems aim for "zero-shot" generalization by using natural language schemas to describe new services. However, these systems often exhibit significant brittleness when faced with linguistic variations in these descriptions. We introduce SGD-X, a benchmark comprising five stylistic variants for the Schema-Guided Dialogue (SGD) dataset. Our evaluation of state-of-the-art models reveals performance drops of up to 28% on these variants. We propose Schema Sensitivity (SS) as a metric for robustness and demonstrate that data augmentation via back-translation can effectively mitigate these issues.

Introduction

In task-oriented dialogue, the industry is shifting toward Schema-Guided Dialogue (SGD). Instead of hard-coding service-specific logic, developers provide a "schema"—a collection of intents (e.g., FindFlight) and slots (e.g., destination_city) described in plain English. Ideally, a model can interpret these descriptions to support a brand-new service without additional training.

Despite the promise of zero-shot generalization, current benchmarks are often too linguistically uniform. In the original SGD dataset, boolean descriptions frequently follow rigid templates, and many intent names in the test set overlap with the training set. This allows models to rely on surface-level pattern matching rather than deep semantic understanding. When these models encounter the diverse writing styles of real-world API developers, they often fail.

The SGD-X Benchmark

To address these limitations, we developed SGD-X, a more rigorous evaluation framework. We employed over 400 crowdsourced workers to generate five stylistic variants ($v_1$ to $v_5$) for every schema in the original SGD dataset.

Examples of schema variants in SGD-X, showing how intent and slot descriptions evolve from the original to more distant paraphrases.

The variants were created using a "blind" paraphrasing technique where workers were asked to invent names based on descriptions or vice versa, ensuring a wide range of linguistic diversity. These variants were then ranked by their linguistic distance from the original, providing a gradient of difficulty for testing.

Methodology

We evaluated two top-performing models: SGP-DST (a BERT-based discriminative model) and T5DST (a T5-based generative model). To measure robustness beyond simple accuracy, we introduced Schema Sensitivity (SS). Based on the Coefficient of Variation, SS measures how much a model’s predictions fluctuate across the five variants. A truly robust model should maintain consistent predictions regardless of whether a slot is named amt or amount_to_transfer.

Additionally, we explored back-translation as a data augmentation strategy. By translating training schemas into intermediate languages (e.g., Japanese or Korean) and back to English, we generated synthetic diversity to help models generalize better.

Key Results

Our evaluation revealed a substantial "generalization illusion." While models performed well on the original "clean" test set, their performance plummeted on the SGD-X variants.

Joint Goal Accuracy (JGA) for the T5DST model across different SGD-X variants, showing a clear performance decline as linguistic distance increases.
  • Performance Drop: Joint Goal Accuracy (JGA) fell by 17.6% for the BERT-based model and 11.9% for the T5-based model on the SGD-X variants compared to the original test set.
  • Worst-Case Scenarios: For the most linguistically distant variant ($v_5$), accuracy dropped by as much as 28%.
  • Model Comparison: The generative T5 model proved significantly more robust and less sensitive to phrasing than the BERT-based model.
  • Augmentation Benefits: Back-translation improved JGA by 8% to 11% and reduced schema sensitivity by up to 17%, though it still trailed behind "Oracle" human-paraphrased data.

Conclusion

SGD-X highlights a critical vulnerability in current schema-guided dialogue systems: linguistic overfitting. Models that appear to generalize well are often just memorizing specific phrasing styles. By incorporating metrics like Schema Sensitivity and benchmarks like SGD-X, researchers can build systems that are truly robust to the diverse ways real-world developers describe their APIs. Future work will extend this analysis to multilingual settings and other components of the dialogue pipeline.

Summary updated

Abstract

Zero/few-shot transfer to unseen services is a critical challenge in task-oriented dialogue research. The Schema-Guided Dialogue (SGD) dataset introduced a paradigm for enabling models to support any service in zero-shot through schemas, which describe service APIs to models in natural language. We explore the robustness of dialogue systems to linguistic variations in schemas by designing SGD-X - a benchmark extending SGD with semantically similar yet stylistically diverse variants for every schema. We observe that two top state tracking models fail to generalize well across schema variants, measured by joint goal accuracy and a novel metric for measuring schema sensitivity. Additionally, we present a simple model-agnostic data augmentation method to improve schema robustness.
BibTeX
@inproceedings{lee2022sgd,
  author = {Harrison Lee and Raghav Gupta and Abhinav Rastogi and Yuan Cao and Bin Zhang and Yonghui Wu},
  title = {{SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21341},
}
Figure