MetaASSIST: Robust Dialogue State Tracking with Meta Learning

Fanghua Ye, Xi Wang, Jie Huang, Shenghui Li, Samuel Stern, Emine Yilmaz

Abstract

Existing dialogue datasets often contain significant noise in their state annotations, which can degrade model training and hinder generalization. While the ASSIST framework was recently introduced to mitigate this by using an auxiliary model to generate pseudo-labels, it relies on a fixed weighting parameter ($\alpha$) to balance vanilla and pseudo-labels. This approach is often suboptimal as it applies a uniform weight across all slots and instances. To address these limitations, we propose MetaASSIST, a meta-learning-based framework that adaptively learns weighting parameters. By implementing three schemes of increasing flexibility—ranging from slot-wise to instance-wise functions—MetaASSIST optimizes weights using a clean validation set as meta-data. Our experiments show that MetaASSIST achieves a state-of-the-art joint goal accuracy of 80.10% on MultiWOZ 2.4.

Introduction

In task-oriented dialogue systems, Dialogue State Tracking (DST) is responsible for maintaining the user's goals throughout a conversation. For instance, if a user asks for a "cheap restaurant in the center," the DST must accurately capture the slots price=cheap and area=center. However, the large-scale datasets required to train these models, such as MultiWOZ, are typically crowdsourced and contain numerous annotation errors. Training on this noisy data leads models to learn and replicate these inaccuracies.

The previous state-of-the-art approach, ASSIST, attempted to solve this by introducing an auxiliary model to generate "pseudo-labels." These labels were then blended with the original noisy labels using a fixed parameter. However, this fixed approach fails to account for the fact that different types of information (slots) have different noise levels, and different conversation turns (instances) vary in complexity. MetaASSIST evolves this concept by making the weighting process dynamic and data-driven.

The MetaASSIST Framework

MetaASSIST replaces the static weighting of ASSIST with learnable functions trained via a meta-learning loop. The framework treats the weighting parameter not as a hyperparameter to be tuned manually, but as a function to be optimized based on the model's performance on a clean validation set.

Three Weighting Schemes

We propose three distinct schemes to convert the weighting parameter into a learnable function:

  1. Scheme One (S1 - Slot-wise): This scheme learns a unique weight for each slot (e.g., hotel-stars vs. taxi-destination). This acknowledges that some slots are inherently more prone to human error than others.
  2. Scheme Two (S2 - Slot-wise & Instance-wise): This scheme uses a Multi-Layer Perceptron (MLP) to determine weights based on the current loss values of both the vanilla and pseudo-labels. This allows the model to decide on a turn-by-turn basis which label is more trustworthy.
  3. Scheme Three (S3 - Decoupled Weights): S3 removes the constraint that weights must sum to one. This provides the flexibility to down-weight both the human label and the pseudo-label if the model detects that both are likely incorrect.

Meta-Learning Optimization

The training process follows a bi-level optimization strategy. In the inner loop, the primary DST model is updated using the weighted combination of labels from the training set. In the outer loop, the meta-learner evaluates the model's performance on a clean validation set and updates the weighting functions to minimize validation loss. This ensures the model learns to prioritize labels that lead to better generalization on high-quality data.

Experimental Results

We evaluated MetaASSIST primarily on MultiWOZ 2.4, a refined version of the MultiWOZ dataset where the validation and test sets have been meticulously cleaned. We also tested on a modified version of MultiWOZ 2.0 to observe performance under higher noise conditions.

Performance Highlights

  • State-of-the-Art Accuracy: MetaASSIST (specifically Scheme S2) achieved a Joint Goal Accuracy (JGA) of 80.10% on MultiWOZ 2.4. This represents a significant milestone, as it is the first time a model has surpassed the 80% threshold on this benchmark.
  • Model Agnostic: The framework was tested across three different DST architectures—SOM-DST, STAR, and AUX-DST. In all cases, MetaASSIST provided a consistent performance boost, demonstrating its versatility as a "plug-and-play" solution for robustness.
  • Domain Robustness: In the "Taxi" domain, which is known for high noise levels in the original data, MetaASSIST outperformed the standard ASSIST baseline by over 4 absolute percentage points.

Discussion and Insights

The success of MetaASSIST stems from the realization that label noise is non-uniform. By allowing the model to recognize that certain slots (like hotel-name) are harder to annotate than others (like hotel-area), the system can effectively filter out human errors. Furthermore, the instance-aware nature of Schemes S2 and S3 allows the model to identify specific dialogue turns where the human annotator may have been distracted or the auxiliary model may have hallucinated.

However, these improvements come with a trade-off in computational cost. The meta-learning process requires multiple passes per training batch, resulting in training times approximately three times longer than standard DST training. Additionally, the framework's reliance on a clean validation set means that the quality of the "meta-data" is critical to the model's success.

Conclusion

MetaASSIST demonstrates that meta-learning is a powerful tool for addressing the pervasive problem of label noise in dialogue systems. By adaptively weighting labels at both the slot and instance levels, MetaASSIST sets a new standard for robustness in Dialogue State Tracking, providing a framework that can be applied to a wide variety of existing model architectures.

Summary updated

Abstract

Existing dialogue datasets contain lots of noise in their state annotations. Such noise can hurt model training and ultimately lead to poor generalization performance. A general framework named ASSIST has recently been proposed to train robust dialogue state tracking (DST) models. It introduces an auxiliary model to generate pseudo labels for the noisy training set. These pseudo labels are combined with vanilla labels by a common fixed weighting parameter to train the primary DST model. Notwithstanding the improvements of ASSIST on DST, tuning the weighting parameter is challenging. Moreover, a single parameter shared by all slots and all instances may be suboptimal. To overcome these limitations, we propose a meta learning-based framework MetaASSIST to adaptively learn the weighting parameter. Specifically, we propose three schemes with varying degrees of flexibility, ranging from slot-wise to both slot-wise and instance-wise, to convert the weighting parameter into learnable functions. These functions are trained in a meta-learning manner by taking the validation set as meta data. Experimental results demonstrate that all three schemes can achieve competitive performance. Most impressively, we achieve a state-of-the-art joint goal accuracy of 80.10% on MultiWOZ 2.4.
BibTeX
@inproceedings{ye2022metaassist,
  author = {Fanghua Ye and Xi Wang and Jie Huang and Shenghui Li and Samuel Stern and Emine Yilmaz},
  title = {{MetaASSIST: Robust Dialogue State Tracking with Meta Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=nf5r_brxjw},
}
Figure