MetaASSIST: Robust Dialogue State Tracking with Meta Learning

Fanghua Ye, Xi Wang, Jie Huang, Shenghui Li, Samuel Stern, Emine Yilmaz

Abstract

Dialogue State Tracking (DST) is a critical component of task-oriented dialogue systems, yet performance is often hindered by noisy, crowdsourced training data. We introduce MetaASSIST, a framework that leverages meta-learning to adaptively weight human-provided "vanilla" labels against model-generated "pseudo-labels." By utilizing learnable functions that operate at both the slot and instance levels, MetaASSIST effectively filters noise and prioritizes high-quality signals. Our approach achieves a new state-of-the-art joint goal accuracy of 80.10% on the MultiWOZ 2.4 benchmark and demonstrates significantly faster convergence compared to existing methods.

Introduction

Dialogue State Tracking (DST) serves as the core intelligence of task-oriented chatbots, mapping user utterances to structured states such as (price=cheap, type=hotel, area=center). However, large-scale datasets like MultiWOZ are typically crowdsourced and contain significant noise, including typos, missing information, and incorrect labels. Training on such data leads to suboptimal model performance. While previous frameworks like ASSIST attempted to mitigate this by using an auxiliary model to generate pseudo-labels, they relied on fixed, manually tuned weights to mix human and model labels. This approach fails to account for varying noise levels across different slots and dialogue turns.

Methodology: Adaptive Label Weighting

MetaASSIST addresses the limitations of fixed weighting by introducing learnable functions that adapt to the specific characteristics of the data. The framework combines pseudo-labels from an auxiliary model (trained on a small, clean dataset) with the original vanilla labels to create an augmented training signal.

The MetaASSIST framework combines pseudo-labels from an auxiliary model with vanilla labels using a learnable weighting parameter α.

The authors propose three schemes for this weighting:

  1. Scheme One (S1 - Slot-wise): Assigns a unique learnable weight to every slot (e.g., hotel-name vs. bus-destination), recognizing that certain categories are more error-prone than others.
  2. Scheme Two (S2 - Instance-wise): Dynamically calculates weights for every dialogue turn using a Multi-Layer Perceptron (MLP) that analyzes the uncertainty (loss) of both label types.
  3. Scheme Three (S3 - Decoupled): Learns two independent weights that do not necessarily sum to one, allowing the model to ignore samples where both labels appear unreliable.

The meta-learning loop uses a small, clean validation set as "meta-data." The model performs a trial training step on noisy data, evaluates performance on the clean set, and updates the weighting functions to minimize validation error.

Key Insights

  • Loss as a Signal: Schemes S2 and S3 use the loss values of labels as features. High loss in a human label signals potential noise, prompting the meta-learner to reduce its influence.
  • Theoretical Foundation: The authors demonstrate mathematically that slot-wise weighting (S1) is guaranteed to perform at least as well as a single shared weight. Their experiments confirmed that instance-level flexibility (S2) provides the optimal balance between performance and stability.
  • Faster Convergence: MetaASSIST doesn't just reach a higher accuracy; it gets there faster. By learning to ignore confusing, noisy samples early on, the primary model focuses on high-quality signals and converges in fewer epochs than the baseline.

Experimental Results

MetaASSIST was evaluated primarily on MultiWOZ 2.4, the most refined version of the dataset where validation and test sets are manually cleaned.

MetaASSIST achieves higher Joint Goal Accuracy (JGA) and faster convergence than the standard ASSIST framework.

Key findings include:

  • State-of-the-Art Performance: MetaASSIST reached 80.10% Joint Goal Accuracy (JGA), becoming the first model to surpass the 80% threshold on MultiWOZ 2.4.
  • Consistency: The framework consistently improved performance across various primary model architectures, including SOM-DST, STAR, and AUX-DST.
  • Domain Gains: In specific domains like "Taxi," MetaASSIST improved accuracy by over 4 absolute percentage points compared to the previous state-of-the-art.

Limitations and Future Work

While effective, MetaASSIST introduces computational overhead; the meta-learning process makes training approximately three times slower than standard methods due to the additional "look-ahead" passes. Additionally, the framework's success depends on the availability of a truly "clean" validation set. Future research may explore using a "committee" of multiple auxiliary models to provide even more robust training signals.

Summary updated

Abstract

Existing dialogue datasets contain lots of noise in their state annotations. Such noise can hurt model training and ultimately lead to poor generalization performance. A general framework named ASSIST has recently been proposed to train robust dialogue state tracking (DST) models. It introduces an auxiliary model to generate pseudo labels for the noisy training set. These pseudo labels are combined with vanilla labels by a common fixed weighting parameter to train the primary DST model. Notwithstanding the improvements of ASSIST on DST, tuning the weighting parameter is challenging. Moreover, a single parameter shared by all slots and all instances may be suboptimal. To overcome these limitations, we propose a meta learning-based framework MetaASSIST to adaptively learn the weighting parameter. Specifically, we propose three schemes with varying degrees of flexibility, ranging from slot-wise to both slot-wise and instance-wise, to convert the weighting parameter into learnable functions. These functions are trained in a meta-learning manner by taking the validation set as meta data. Experimental results demonstrate that all three schemes can achieve competitive performance. Most impressively, we achieve a state-of-the-art joint goal accuracy of 80.10% on MultiWOZ 2.4.
BibTeX
@inproceedings{ye2022metaassist,
  author = {Fanghua Ye and Xi Wang and Jie Huang and Shenghui Li and Samuel Stern and Emine Yilmaz},
  title = {{MetaASSIST: Robust Dialogue State Tracking with Meta Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=nf5r_brxjw},
}
Figure