Improving Dialogue State Tracking with Turn-based Loss Function and Sequential Data Augmentation

Jarana Manotumruksa, Jeff Dalton, Edgar Meij, Emine YilmazEMNLP 2021 · OpenReview

Abstract

While state-of-the-art Dialogue State Tracking (DST) models show promising results, all of them rely on a traditional cross-entropy loss function during the training process, which may not be optimal for improving the joint goal accuracy. Although several approaches recently propose augmenting the training set by copying user utterances and replacing the real slot values with other possible or even similar values, they are not effective at improving the performance of existing DST models. To address these challenges, we propose a Turn-based Loss Function (TLF) that penalises the model if it inaccurately predicts a slot value at the early turns more so than in later turns in order to improve joint goal accuracy. We also propose a simple but effective Sequential Data Augmentation (SDA) algorithm to generate more complex user utterances and system responses to effectively train existing DST models. Experimental results on two standard DST benchmark collections demonstrate that our proposed TLF and SDA techniques significantly improve the effectiveness of the state-of-the-art DST model by approximately 7-8% relative reduction in error and achieves a new state-of-the-art joint goal accuracy with 59.50 and 54.90 on MultiWOZ2.1 and MultiWOZ2.2, respectively.

Introduction

In task-oriented dialogue systems, Dialogue State Tracking (DST) is critical for maintaining a representation of the user's goals throughout a conversation. The standard metric for evaluation is Joint Goal Accuracy (JGA), which requires all slot-value pairs to be correctly identified in a given turn. A significant challenge in current DST models is the "error cascade" effect: an incorrect prediction in an early turn often propagates through the rest of the dialogue, leading to a JGA of zero for all subsequent turns.

Example of error propagation in dialogue state prediction. An error in the first turn (Prediction 1) leads to a failure in Joint Goal Accuracy (JGA: 0) for the entire session.

Methodology

To mitigate these issues, we introduce two novel techniques integrated into the TripPy model architecture, resulting in TripPy-TS.

Turn-based Loss Function (TLF)

Traditional training uses a flat cross-entropy loss. We propose a time-decaying penalty that prioritizes accuracy in early turns. The loss for turn $t$ is weighted by a factor $[1 + (T - t) * \lambda]$, where $T$ is the total dialogue length and $\lambda$ is a weight parameter. This encourages the model to establish a correct state early on, preventing downstream failures that ruin the JGA.

Sequential Data Augmentation (SDA)

Existing datasets often contain sparse turns with few active slots. SDA generates denser training examples by concatenating a turn $t$ with its successors. This forces the model to learn from more complex utterances containing multiple slot updates, improving its robustness to multi-intent user requests without needing external generative models or databases.

Experimental Results

We evaluated TripPy-TS on MultiWOZ 2.1 and 2.2. The model achieved a new state-of-the-art JGA of 59.50% on MultiWOZ 2.1 and 54.90% on MultiWOZ 2.2, representing a 7-8% relative error reduction over the baseline.

A key finding is the model's performance on complex turns. As shown in the figure below, TripPy-TS significantly outperforms the baseline as the number of active slots per turn increases. On turns with 4 active slots, TripPy-TS outperformed the baseline by a staggering 23-34% in relative error reduction.

Joint Goal Accuracy comparison between TripPy and TripPy-TS across different numbers of active slots per turn on MultiWOZ 2.1 and 2.2.

Conclusion

By addressing the temporal importance of turns and the sparsity of training data, TLF and SDA provide a robust framework for improving DST performance. Future work will explore dynamic penalty weights based on dialogue complexity and combining SDA with value-substitution methods for further generalization.

Summary updated

Abstract

While state-of-the-art Dialogue State Tracking (DST) models show promising results, all of them rely on a traditional cross-entropy loss function during the training process, which may not be optimal for improving the joint goal accuracy. Although several approaches recently propose augmenting the training set by copying user utterances and replacing the real slot values with other possible or even similar values, they are not effective at improving the performance of existing DST models. To address these challenges, we propose a Turn-based Loss Function (TLF) that penalises the model if it inaccurately predicts a slot value at the early turns more so than in later turns in order to improve joint goal accuracy. We also propose a simple but effective Sequential Data Augmentation (SDA) algorithm to generate more complex user utterances and system responses to effectively train existing DST models. Experimental results on two standard DST benchmark collections demonstrate that our proposed TLF and SDA techniques significantly improve the effectiveness of the state-of-the-art DST model by approximately 7-8% relative reduction in error and achieves a new state-of-the-art joint goal accuracy with 59.50 and 54.90 on MultiWOZ2.1 and MultiWOZ2.2, respectively.
BibTeX
@inproceedings{manotumruksa2021improving,
  author = {Jarana Manotumruksa and Jeff Dalton and Edgar Meij and Emine Yilmaz},
  title = {{Improving Dialogue State Tracking with Turn-based Loss Function and Sequential Data Augmentation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=Cp3gXq3PlZ},
}
Figure