Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics

Hyundong Cho, Chinnadhurai Sankar, Christopher Lin, Kaushik Ram Sadagopan, Shahin Shayandeh, Aslı Çelikyılmaz, Jonathan May, Ahmad Beirami

Abstract

Recent works that revealed the vulnerability of dialogue state tracking (DST) models to distributional shifts have made holistic comparisons on robustness and qualitative analyses increasingly important for understanding their relative performance. We present our findings from standardized and comprehensive DST diagnoses, which have previously been sparse and uncoordinated, using our toolkit, CheckDST, a collection of robustness tests and failure mode analytics. We discover that different classes of DST models have clear strengths and weaknesses, where generation models are more promising for handling language variety while span-based classification models are more robust to unseen entities. Prompted by this discovery, we also compare checkpoints from the same model and find that the standard practice of selecting checkpoints using validation loss/accuracy is prone to overfitting and each model class has distinct patterns of failure. Lastly, we demonstrate how our diagnoses motivate a pre-finetuning procedure with non-dialogue data that offers comprehensive improvements to generation models by alleviating the impact of distributional shifts through transfer learning.

Introduction

Dialogue State Tracking (DST) serves as the core component of task-oriented dialogue systems, responsible for maintaining the "belief state"—a structured representation of user constraints (e.g., food: Italian). Despite significant progress, evaluation has largely relied on Joint Goal Accuracy (JGA), a "black box" metric that fails to capture model robustness or specific failure modes. A model might achieve high JGA by memorizing training patterns but collapse when faced with minor linguistic variations, such as speech disfluencies or paraphrasing. This work introduces CheckDST, a diagnostic toolkit designed to provide fine-grained insights into DST model performance and robustness.

The CheckDST Diagnostic Toolkit

CheckDST evaluates models using consistent JGA (cJGA), which requires a model to correctly predict the state for both an original utterance and its perturbed counterpart. The toolkit includes several perturbation types:

  • Language Variety: Paraphrasing and synonyms.
  • Speech Disfluency: Utterances containing fillers (e.g., "uh") or self-corrections.
  • Unseen Entities: Scrambled or novel entity names not present in the training set.
  • Coreference Resolution: Complex scenarios requiring the resolution of pronouns across turns.

The primary benchmark for these evaluations is MultiWOZ (2.1 and 2.3), the standard for multi-domain dialogue.

The Strengths Divide and the Validation Trap

Our diagnostics reveal a fundamental divide between model architectures. Classification-based models (SCLS), which extract spans from the input, excel at handling unseen entities because they are inherently designed to copy text. In contrast, generation-based models (GEN) are more adept at handling language variety but often "hallucinate" or fail when encountering unfamiliar names.

Furthermore, we identify a "Validation Trap." Standard practice involves selecting model checkpoints based on peak validation JGA. However, our analysis shows that robustness to perturbations often peaks early in training and subsequently declines even as JGA continues to improve. This suggests that models increasingly overfit to the specific linguistic patterns of the training set at the expense of generalizability.

Robustness metrics vs. training epochs for TripPy and BART-DST.

PrefineDST: Enhancing Robustness through Transfer Learning

To address these weaknesses, we propose PrefineDST, a training strategy that incorporates an intermediate "pre-finetuning" phase. By training on diverse non-dialogue tasks—such as SQuAD (reading comprehension), WikiSQL (structured data), and MSR Paraphrase—the model acquires fundamental skills like text copying and coreference resolution before being exposed to dialogue-specific data.

Key Results

PrefineDST emerges as a more well-rounded model, bridging the gap between classification and generation strengths. It improves the baseline JGA of generative models while significantly boosting robustness across all CheckDST categories.

Comprehensive diagnostic results comparing classification (SCLS) and generation (GEN) models.

As shown in the results table, SCLS models like TripPy maintain a significant lead in handling unseen entities (NED cJGA), while GEN models generally perform better on paraphrasing tasks. PrefineDST successfully mitigates many of the generative baseline's weaknesses, achieving a JGA of 55.7% compared to the 52.5% baseline.

Conclusion

Comprehensive diagnostics through CheckDST reveal that high JGA often masks significant vulnerabilities. By understanding the inherent trade-offs between model classes and the risks of standard checkpoint selection, researchers can develop more robust systems. PrefineDST demonstrates that general-purpose linguistic skills are highly transferable to the DST task, offering a path toward more reliable AI assistants. Future work will explore these dynamics in multi-lingual settings and investigate methods to further reduce hallucinations in generative architectures.

Summary updated

Abstract

Recent works that revealed the vulnerability of dialogue state tracking (DST) models to distributional shifts have made holistic comparisons on robustness and qualitative analyses increasingly important for understanding their relative performance. We present our findings from standardized and comprehensive DST diagnoses, which have previously been sparse and uncoordinated, using our toolkit, CheckDST, a collection of robustness tests and failure mode analytics. We discover that different classes of DST models have clear strengths and weaknesses, where generation models are more promising for handling language variety while span-based classification models are more robust to unseen entities. Prompted by this discovery, we also compare checkpoints from the same model and find that the standard practice of selecting checkpoints using validation loss/accuracy is prone to overfitting and each model class has distinct patterns of failure. Lastly, we demonstrate how our diagnoses motivate a pre-finetuning procedure with non-dialogue data that offers comprehensive improvements to generation models by alleviating the impact of distributional shifts through transfer learning.
BibTeX
@inproceedings{cho2022know,
  author = {Hyundong Cho and Chinnadhurai Sankar and Christopher Lin and Kaushik Ram Sadagopan and Shahin Shayandeh and Aslı Çelikyılmaz and Jonathan May and Ahmad Beirami},
  title = {{Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.391},
}
Figure