How to Stop an Avalanche? JoDeM: Joint Decision Making through Compare and Contrast for Dialog State Tracking

Haoming Wang, Wang XinEMNLP 2022 · DOI

Introduction

In Task-Oriented Dialogue (TOD) systems, Dialog State Tracking (DST) is the core engine that maintains the user's goals and constraints throughout a conversation. Most contemporary DST models employ a turn-by-turn prediction scheme, where the state at turn $i$ is predicted based on the current utterance and the predicted state from turn $i-1$.

This dependency creates a significant vulnerability known as the Avalanche Phenomenon. Because models are typically trained on ground-truth previous states, they are ill-equipped to handle their own errors in a real-world setting. A single incorrect prediction early in a dialogue can propagate through subsequent turns, leading to a rapid decline in system accuracy as errors accumulate.

The JoDeM Architecture

To mitigate the avalanche effect, the authors propose JoDeM (Joint Decision Making), a framework that decouples feature extraction from the state update process. JoDeM is organized into three primary stages:

  1. Utterance Encoding: Utilizing a pre-trained BERT backbone, the system generates contextual embeddings for the dialogue history and the current user utterance.
  2. Joint Decision Making: This stage consists of parallel modules that extract information independently of the previous state's specific values. Key modules include Domain Update, Slot Gate, Type Prediction, Span Prediction, and Co-reference Classification. This parallel structure ensures that the extraction process is not poisoned by prior errors. Additionally, a diagonal coefficient matrix ($\gamma$) is used to handle domain transitions, allowing the previous domain to act as a bias that is only overridden when a new domain is detected.
  3. Dialogue State Update: This module implements a "compare and contrast" mechanism. It evaluates the newly extracted information against the existing dialogue state and only updates the state when the Joint Decision stage provides high-confidence new information.
Figure 1: The JoDeM architecture, illustrating the flow from utterance encoding to joint decision making and the final state update.

Self-Correction Training

JoDeM incorporates a reinforcement-based training strategy. When the model's forward pass correctly identifies a value, the output is forced to the ground-truth label. This prevents backpropagation from destabilizing successful parameters, effectively teaching the model to maintain correct states once they are reached.

Experimental Evaluation

The model was evaluated on MultiWOZ 2.3, a refined version of the multi-domain Wizard-of-Oz dataset. To measure robustness against error propagation, the authors introduced the Avalanche Coefficient ($\alpha$), where a value of 1.0 indicates perfect immunity to the avalanche effect.

Results and Discussion

JoDeM achieves a new state-of-the-art performance on the MultiWOZ 2.3 benchmark, significantly outperforming existing models in both accuracy and robustness.

  • Joint Accuracy: JoDeM reached a Joint Accuracy of 74.9%, a substantial improvement over previous benchmarks like TripPy (63.2%) and ReDST (64.0%).
  • Robustness: JoDeM achieved an Avalanche Coefficient of 0.979, compared to the 0.93-0.95 range typical of standard turn-by-turn models. This indicates that JoDeM is far more successful at preventing minor errors from snowballing into total system failure.
Table 1: Performance comparison of JoDeM against baseline models on the MultiWOZ 2.3 dataset.

Qualitative analysis suggests that the decoupling of extraction and updates provides a safety net; for instance, if the Slot Gate incorrectly flags a change, the Span Detection module can still prevent a wrongful update by pointing to a null token.

Conclusion

JoDeM demonstrates that the Avalanche Phenomenon in DST can be effectively managed by separating the extraction of dialogue features from the maintenance of the dialogue state. By employing a "compare and contrast" logic, JoDeM provides a robust framework for building more reliable task-oriented dialogue systems. Future work will focus on optimizing the model's computational efficiency and testing its generalization capabilities in zero-shot multi-domain scenarios.

Summary updated

BibTeX
@inproceedings{wang2022how,
  author = {Haoming Wang and Wang Xin},
  title = {{How to Stop an Avalanche? JoDeM: Joint Decision Making through Compare and Contrast for Dialog State Tracking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.521},
}
Figure