ASSIST: Towards Label Noise-Robust Dialogue State Tracking

Fanghua Ye, Yue Feng, Emine Yilmaz

Abstract

The MultiWOZ 2.0 dataset has greatly boosted the research on dialogue state tracking (DST). However, substantial noise has been discovered in its state annotations. Such noise brings about huge challenges for training DST models robustly. Although several refined versions, including MultiWOZ 2.1-2.4, have been published recently, there are still lots of noisy labels, especially in the training set. Besides, it is costly to rectify all the problematic annotations. In this paper, instead of improving the annotation quality further, we propose a general framework, named ASSIST (lAbel noiSe-robuSt dIalogue State Tracking), to train DST models robustly from noisy labels. ASSIST first generates pseudo labels for each sample in the training set by using an auxiliary model trained on a small clean dataset, then puts the generated pseudo labels and vanilla noisy labels together to train the primary model. Experimental results demonstrate that ASSIST improves the joint goal accuracy of DST by up to 28.16% on MultiWOZ 2.0 and 8.41% on MultiWOZ 2.4, compared to using only the vanilla noisy labels.

Introduction: The Challenge of Noisy Labels in DST

Dialogue State Tracking (DST) serves as the core component of task-oriented chatbots, responsible for tracking user requirements—such as hotel preferences or cinema locations—across a conversation. However, the field’s benchmark dataset, MultiWOZ 2.0, is notoriously noisy; subsequent revisions revealed that over 40% of its state annotations were incorrect or inconsistent.

While cleaner versions like MultiWOZ 2.1–2.4 have been released, these typically only rectify validation and test sets. The massive training sets remain riddled with errors because manual re-annotation is prohibitively expensive. This "label noise" confuses models during training, leading to sub-optimal performance. The ASSIST framework addresses this by designing a training process that handles imperfect data rather than attempting to manually clean every label.

The ASSIST Framework

The authors propose ASSIST (lAbel noiSe robuSt dIalogue State Tracking), which utilizes a two-stage training pipeline to mitigate the impact of noise.

First, an Auxiliary Model is trained on a small, manually verified "clean" dataset (typically the rectified validation set). This model uses a BERT-based architecture with a slot-attention mechanism to match dialogue context with specific slot-value pairs. Once trained, this auxiliary model is run over the large, noisy training set to generate "pseudo-labels"—its own high-confidence predictions of the correct states.

The architecture of the auxiliary model in the ASSIST framework, utilizing BERT and Slot Attention.

Second, the Primary Model (which can be any existing DST architecture) is trained using a weighted combination of the original "vanilla" noisy labels and the newly generated pseudo-labels. This combined loss function is governed by a hyperparameter $\alpha$, allowing the model to learn from the original data while imitating the cleaner predictions of the auxiliary model.

Balancing Noisy and Pseudo Labels

A critical component of ASSIST is the weighting parameter $\alpha$. The researchers found that the optimal balance between noisy vanilla labels and cleaner pseudo-labels varies by dataset. As shown in the sensitivity analysis, the Joint Goal Accuracy peaks when $\alpha$ is around 0.4 to 0.6, demonstrating that neither source is sufficient on its own. The pseudo-labels and noisy labels often fail in different ways; by combining them, the primary model can "triangulate" the true state more accurately.

Joint Goal Accuracy as a function of the weighting parameter α for MultiWOZ 2.0 and 2.4.

Key Results and Insights

The results were dramatic, particularly on the noisier MultiWOZ 2.0 dataset. The Joint Goal Accuracy (JGA) for the SOM-DST model jumped from 45.14% to 70.83% when using the ASSIST framework. Even on the cleaner MultiWOZ 2.4, JGA improved significantly (e.g., from 73.62% to 79.41% for the STAR model).

Key takeaways from the study include:

  • The Power of Small Data: A small, high-quality "anchor" (like a 1,000-dialogue validation set) is sufficient to bootstrap an auxiliary model that filters noise for the entire training set.
  • Model Agnosticism: ASSIST is a training framework, not a specific architecture. It consistently improved performance across different base models like SOM-DST and STAR.
  • Robustness: Qualitative analysis confirmed that pseudo-labels successfully captured missing information and corrected "hallucinated" values present in the original noisy annotations.

Conclusion

ASSIST demonstrates that model-based label cleaning is a viable and highly effective alternative to manual re-annotation. While the framework currently assumes access to a small clean validation set and uses a static weighting parameter, future work may explore adaptive weighting that adjusts based on the estimated noise level of specific dialogue slots. Despite these areas for growth, ASSIST provides a significant leap forward in building robust conversational AI from real-world, imperfect data.

Summary updated

Abstract

The MultiWOZ 2.0 dataset has greatly boosted the research on dialogue state tracking (DST). However, substantial noise has been discovered in its state annotations. Such noise brings about huge challenges for training DST models robustly. Although several refined versions, including MultiWOZ 2.1-2.4, have been published recently, there are still lots of noisy labels, especially in the training set. Besides, it is costly to rectify all the problematic annotations. In this paper, instead of improving the annotation quality further, we propose a general framework, named ASSIST (lAbel noiSe-robuSt dIalogue State Tracking), to train DST models robustly from noisy labels. ASSIST first generates pseudo labels for each sample in the training set by using an auxiliary model trained on a small clean dataset, then puts the generated pseudo labels and vanilla noisy labels together to train the primary model. We show the validity of ASSIST theoretically. Experimental results also demonstrate that ASSIST improves the joint goal accuracy of DST by up to $28.16\%$ on MultiWOZ 2.0 and $8.41\%$ on MultiWOZ 2.4, compared to using only the vanilla noisy labels.
BibTeX
@inproceedings{ye2022assist,
  author = {Fanghua Ye and Yue Feng and Emine Yilmaz},
  title = {{ASSIST: Towards Label Noise-Robust Dialogue State Tracking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=XjGtj-fX6u0},
}
Figure