Abstract
Existing dialogue datasets often contain significant noise in their state annotations, which can degrade model training and hinder generalization. While the ASSIST framework was recently introduced to mitigate this by using an auxiliary model to generate pseudo-labels, it relies on a fixed weighting parameter ($\alpha$) to balance vanilla and pseudo-labels. This approach is often suboptimal as it applies a uniform weight across all slots and instances. To address these limitations, we propose MetaASSIST, a meta-learning-based framework that adaptively learns weighting parameters. By implementing three schemes of increasing flexibility—ranging from slot-wise to instance-wise functions—MetaASSIST optimizes weights using a clean validation set as meta-data. Our experiments show that MetaASSIST achieves a state-of-the-art joint goal accuracy of 80.10% on MultiWOZ 2.4.
Introduction
In task-oriented dialogue systems, Dialogue State Tracking (DST) is responsible for maintaining the user's goals throughout a conversation. For instance, if a user asks for a "cheap restaurant in the center," the DST must accurately capture the slots price=cheap and area=center. However, the large-scale datasets required to train these models, such as MultiWOZ, are typically crowdsourced and contain numerous annotation errors. Training on this noisy data leads models to learn and replicate these inaccuracies.
The previous state-of-the-art approach, ASSIST, attempted to solve this by introducing an auxiliary model to generate "pseudo-labels." These labels were then blended with the original noisy labels using a fixed parameter. However, this fixed approach fails to account for the fact that different types of information (slots) have different noise levels, and different conversation turns (instances) vary in complexity. MetaASSIST evolves this concept by making the weighting process dynamic and data-driven.
The MetaASSIST Framework
MetaASSIST replaces the static weighting of ASSIST with learnable functions trained via a meta-learning loop. The framework treats the weighting parameter not as a hyperparameter to be tuned manually, but as a function to be optimized based on the model's performance on a clean validation set.
Three Weighting Schemes
We propose three distinct schemes to convert the weighting parameter into a learnable function:
- Scheme One (S1 - Slot-wise): This scheme learns a unique weight for each slot (e.g.,
hotel-starsvs.taxi-destination). This acknowledges that some slots are inherently more prone to human error than others. - Scheme Two (S2 - Slot-wise & Instance-wise): This scheme uses a Multi-Layer Perceptron (MLP) to determine weights based on the current loss values of both the vanilla and pseudo-labels. This allows the model to decide on a turn-by-turn basis which label is more trustworthy.
- Scheme Three (S3 - Decoupled Weights): S3 removes the constraint that weights must sum to one. This provides the flexibility to down-weight both the human label and the pseudo-label if the model detects that both are likely incorrect.
Meta-Learning Optimization
The training process follows a bi-level optimization strategy. In the inner loop, the primary DST model is updated using the weighted combination of labels from the training set. In the outer loop, the meta-learner evaluates the model's performance on a clean validation set and updates the weighting functions to minimize validation loss. This ensures the model learns to prioritize labels that lead to better generalization on high-quality data.
Experimental Results
We evaluated MetaASSIST primarily on MultiWOZ 2.4, a refined version of the MultiWOZ dataset where the validation and test sets have been meticulously cleaned. We also tested on a modified version of MultiWOZ 2.0 to observe performance under higher noise conditions.
Performance Highlights
- State-of-the-Art Accuracy: MetaASSIST (specifically Scheme S2) achieved a Joint Goal Accuracy (JGA) of 80.10% on MultiWOZ 2.4. This represents a significant milestone, as it is the first time a model has surpassed the 80% threshold on this benchmark.
- Model Agnostic: The framework was tested across three different DST architectures—SOM-DST, STAR, and AUX-DST. In all cases, MetaASSIST provided a consistent performance boost, demonstrating its versatility as a "plug-and-play" solution for robustness.
- Domain Robustness: In the "Taxi" domain, which is known for high noise levels in the original data, MetaASSIST outperformed the standard ASSIST baseline by over 4 absolute percentage points.
Discussion and Insights
The success of MetaASSIST stems from the realization that label noise is non-uniform. By allowing the model to recognize that certain slots (like hotel-name) are harder to annotate than others (like hotel-area), the system can effectively filter out human errors. Furthermore, the instance-aware nature of Schemes S2 and S3 allows the model to identify specific dialogue turns where the human annotator may have been distracted or the auxiliary model may have hallucinated.
However, these improvements come with a trade-off in computational cost. The meta-learning process requires multiple passes per training batch, resulting in training times approximately three times longer than standard DST training. Additionally, the framework's reliance on a clean validation set means that the quality of the "meta-data" is critical to the model's success.
Conclusion
MetaASSIST demonstrates that meta-learning is a powerful tool for addressing the pervasive problem of label noise in dialogue systems. By adaptively weighting labels at both the slot and instance levels, MetaASSIST sets a new standard for robustness in Dialogue State Tracking, providing a framework that can be applied to a wide variety of existing model architectures.
Summary updated