Abstract
Zero/few-shot transfer to unseen services is a critical challenge in task-oriented dialogue research. The Schema-Guided Dialogue (SGD) dataset introduced a paradigm for enabling models to support any service in zero-shot through schemas, which describe service APIs to models in natural language. We explore the robustness of dialogue systems to linguistic variations in schemas by designing SGD-X - a benchmark extending SGD with semantically similar yet stylistically diverse variants for every schema. We observe that two top state tracking models fail to generalize well across schema variants, measured by joint goal accuracy and a novel metric for measuring schema sensitivity. Additionally, we present a simple model-agnostic data augmentation method to improve schema robustness.
Introduction
The primary goal of modern task-oriented dialogue systems is zero-shot generalization: the ability to support new services without requiring service-specific training data. The Schema-Guided Dialogue (SGD) paradigm facilitates this by providing models with a "schema"—a set of natural language descriptions for intents and slots. However, the original SGD dataset is linguistically uniform, often featuring repetitive phrasing that allows models to overfit to specific keywords rather than understanding the underlying semantics. This paper introduces SGD-X to stress-test the robustness of these systems against stylistic variations in schema descriptions.
The SGD-X Benchmark
To create a more rigorous evaluation, we developed SGD-X, an extension of the SGD dataset featuring five stylistic variants ($v1$ to $v5$) for every schema. These variants were generated through crowdsourced paraphrasing involving over 400 workers. To ensure diversity, workers used "blind" paraphrasing, where they were asked to invent names based on descriptions or vice versa without seeing the original labels.
As shown in the figure above, the variants range from minor tweaks ($v1$) to more distant, "wild" phrasing ($v5$). This diversity exposes a critical flaw in the original SGD dataset: in the baseline test set, 71% of "unseen" intent names actually appeared in the training set. In SGD-X, this overlap drops to just 2%, forcing models to rely on semantic understanding rather than word recognition.
Experimental Setup and Metrics
We evaluated two state-of-the-art Dialogue State Tracking (DST) models: the BERT-based SGP-DST and the generative T5-based T5DST. Beyond standard Joint Goal Accuracy (JGA), we introduced a new metric, Schema Sensitivity (SS), to measure how much a model's predictions fluctuate when the schema's wording changes. A truly robust model should maintain consistent outputs regardless of whether a field is labeled "amount" or "cash_to_transfer."
Key Findings and Results
Our experiments revealed a significant "robustness gap" in current models. When tested on SGD-X variants, the BERT-based model's JGA dropped by 17.6%, while the T5-based model saw an 11.9% decline. The T5 model proved generally more robust and less sensitive, likely due to its extensive pre-training on diverse text corpora.
As visualized in the charts above, performance consistently degrades as the linguistic distance from the original schema increases. The models struggle particularly with boolean descriptions; if a model is trained only on descriptions starting with "Whether...", it often fails when encountering a variant starting with "Boolean flag indicating...".
To address this, we tested a model-agnostic data augmentation technique using back-translation. By translating training schemas into other languages and back to English, we introduced synthetic variety during training. This simple fix improved JGA on the SGD-X benchmark by 8% for the BERT model and 11% for the T5 model. Furthermore, an "Oracle" setup—training on human-paraphrased schemas—showed performance gains of up to 33%, indicating substantial room for improvement in training data generation.
Conclusion
SGD-X reveals that state-of-the-art dialogue systems are highly sensitive to the linguistic style of service schemas. While back-translation offers a partial remedy, it cannot yet match the nuance of human creativity. Future work should expand this robustness analysis to other components of the dialogue pipeline, such as Natural Language Generation, and explore models capable of handling the even messier documentation found in real-world APIs.
Summary updated