CheckDST: Measuring Real-World Generalization of Dialogue State Tracking Performance

Hyundong Justin Cho, Chinnadhurai Sankar, Christopher Lin, Kaushik Ram Sadagopan, Shahin Shayandeh, Asli Celikyilmaz, Jonathan May, Ahmad Beirami

Abstract

Recent works that revealed the vulnerability of dialogue state tracking (DST) models to distributional shifts have made holistic comparisons on robustness and qualitative analyses increasingly important for understanding their relative performance. We present our findings from standardized and comprehensive DST diagnoses using our toolkit, CheckDST, a collection of robustness tests and failure mode analytics. We discover that different classes of DST models have clear strengths and weaknesses: generation models are more promising for handling language variety, while span-based classification models are more robust to unseen entities. We also demonstrate how our diagnoses motivate a pre-finetuning procedure, PrefineDST, which offers comprehensive improvements to generation models by alleviating the impact of distributional shifts through transfer learning.

Introduction

Dialogue State Tracking (DST) is the task of identifying a user's requirements and updating the "state" so a system can query an API. While models have reached high scores on benchmarks like MultiWOZ, they are notoriously fragile. A simple paraphrase or a slight speech disfluency can cause them to fail. Current evaluation relies almost entirely on Joint Goal Accuracy (JGA), a single metric that fails to explain why a model fails or how it will behave under real-world distributional shifts. This work introduces CheckDST to move beyond simple accuracy rankings toward a deeper understanding of model strengths and weaknesses.

The CheckDST Toolkit

CheckDST is a diagnostic toolkit that subjects models to five types of perturbations: Paraphrase, Speech Disfluency, Unseen Entities, Coreference, and Hallucination. To measure these, we introduce consistent JGA (cJGA). Unlike standard metrics, cJGA only awards a point if the model is correct on both the original and the perturbed version of a turn, ensuring the model's performance is robust rather than coincidental.

Table 1: Examples of the linguistic perturbations and failure modes captured in CheckDST.

Analysis of Model Classes

Our diagnosis reveals a clear divergence between model types:

  • Classification-based models (e.g., TripPy) are excellent at handling unseen names because they "copy" spans of text, but they struggle with language variety.
  • Generation models (e.g., BART-DST) are much better at handling paraphrases but are prone to "hallucinating" values not present in the text and fail when names are unfamiliar.

We also identify a "validation trap": robustness metrics often peak early in training and then decline as the model overfits to the training set's specific phrasing, even as standard JGA continues to improve.

PrefineDST: Multi-task Pre-finetuning

To address these weaknesses, we developed PrefineDST. Instead of training only on dialogue data, we first "pre-finetune" a BART model on a variety of non-dialogue NLP tasks—such as SQuAD for extraction, QQP for paraphrasing, and WSC for coreference—using natural language instructions. This teaches the model general linguistic skills before it is applied to the specific DST task.

Results

The CheckDST diagnosis confirms that PrefineDST is a more well-rounded model. By learning general linguistic skills first, it achieves higher JGA (55.7%) than other generation baselines and superior robustness, hitting the highest scores in Paraphrase (46.1 PI cJGA) and Speech Disfluency (41.8 SDI cJGA) metrics. It also shows significantly improved factuality compared to standard generation models.

Table 2: Quantitative comparison of different model classes across CheckDST metrics. Green and red highlights indicate relative strengths and weaknesses.

Conclusion

CheckDST provides a richer summary of model performance compared to standard benchmarks. Our findings highlight the inherent trade-offs in current DST architectures and demonstrate that multi-task pre-finetuning is a promising path toward more robust and generalizable dialogue systems.

Summary updated

Abstract

Recent works that revealed the vulnerability of dialogue state tracking (DST) models to distributional shifts have made holistic comparisons on robustness and qualitative analyses increasingly important for understanding their relative performance. We present our findings from standardized and comprehensive DST diagnoses, which have previously been sparse and uncoordinated, using our toolkit, CheckDST, a collection of robustness tests and failure mode analytics. We discover that different classes of DST models have clear strengths and weaknesses, where generation models are more promising for handling language variety while span-based classification models are more robust to unseen entities. Prompted by this discovery, we also compare checkpoints from the same model and find that the standard practice of selecting checkpoints using validation loss/accuracy is prone to overfitting and each model class has distinct patterns of failure. Lastly, we demonstrate how our diagnoses motivate a pre-finetuning procedure with non-dialogue data that offers comprehensive improvements to generation models by alleviating the impact of distributional shifts through transfer learning.
BibTeX
@inproceedings{cho2022checkdst,
  author = {Hyundong Justin Cho and Chinnadhurai Sankar and Christopher Lin and Kaushik Ram Sadagopan and Shahin Shayandeh and Asli Celikyilmaz and Jonathan May and Ahmad Beirami},
  title = {{CheckDST: Measuring Real-World Generalization of Dialogue State Tracking Performance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.391},
}
Figure