How to Tame Your Data: Data Augmentation for Dialog State Tracking

Adam Summerville, Jordan Hashemi, James Ryan, William FergusonACL 2020

TL;DR

This paper tackles the challenge of Dialog State Tracking (DST) systems struggling with the virtually limitless vocabulary of real-world conversations. They introduce a novel data augmentation technique that intelligently injects diverse, real-world slot values into training data, significantly improving model robustness and achieving state-of-the-art results on challenging benchmarks.

What Problem Does This Paper Address?

Modern conversational AI systems rely on Dialog State Tracking (DST) to understand and remember user requests. Imagine ordering a pizza: the system needs to track the pizza type, toppings, size, and delivery address. The challenge is that the possible values for these "slots" are practically infinite – think of all the possible pizza toppings or street names! This means DST models often encounter "out-of-vocabulary" (OOV) words during real-time use that they never saw during training, leading to errors. Existing methods like simply masking unknown words or using generic tokens don't fully address this, as they don't teach the model to actually handle diverse vocabulary. This paper aims to create a DST system that can gracefully handle unseen vocabulary and adapt to new domains with minimal architectural changes.

Methods

The core of this paper is a two-pronged approach:

  1. Language Model-Based Transformer Architecture: Instead of relying on specialized output heads for each slot, the authors leverage a fine-tuned RoBERTa Transformer. The input utterance is augmented with a sequence of all available slots, and the model is trained to predict two things:
  2. Utterance-level slot activation: Whether a slot is present in the current utterance (e.g., "movie" is active).
  3. Token-level slot filling: For each token in the utterance, which slot it belongs to (e.g., "Transformers" belongs to the "movie" slot). This textual, end-to-end approach allows the architecture to scale to new intents and slots without modification.

  4. Targeted Data Augmentation: This is the paper's main innovation. Instead of generic augmentation, they propose a systematic process:

  5. Identify Problem Slots: Analyze evaluation errors to pinpoint which slots the model struggles with (e.g., movie titles, restaurant names).
  6. Augment with Real-World Corpora: For these problematic slots, they find external corpora of relevant values (e.g., Wikipedia lists of movie titles, lists of restaurant names). These real values are then randomly inserted into the training data at a significant rate (50% in their experiments).

This augmentation has two key effects: it exposes the model to a wider variety of slot values and structures, and it makes the dialogs more challenging, forcing the model to learn to track user goals more robustly, even when values change.

Datasets

The authors evaluate their approach on two existing datasets:

  • Sim-R: A dataset focused on restaurant reservations.
  • Sim-M: A dataset focused on movie ticket bookings.

These datasets are noted as being of high quality but with room for improvement, particularly regarding the open-class nature of slots like restaurant names and movie titles.

Key Insights and Ideas

  • Beyond Masking: Simply masking OOV words or using special tokens is insufficient. Models need to learn to handle diverse, real-world values.
  • Augmentation with Real Data is Powerful: Injecting actual, diverse slot values (like movie titles or restaurant names) into training data is more effective than generic augmentation or just masking.
  • Challenging the Model Improves Robustness: Making the dialogs harder by introducing varied slot values forces the model to learn more robust tracking mechanisms, akin to a user who frequently changes their mind.
  • Unified Architecture for Extensibility: Using a language model-based Transformer allows the system to adapt to new domains and schemas with minimal architectural changes, making it highly extensible.

Key Results

The proposed RoBERTa-based model with targeted data augmentation achieved state-of-the-art results on both Sim-M and Sim-R datasets.

  • ROBERTa-Separate (separate models for Sim-M and Sim-R) showed a 4.1% and 3.1% improvement over the previous state-of-the-art on Sim-M and Sim-R, respectively.
  • ROBERTa-Combined (a single model trained on both datasets) also performed very well, with significant gains, especially on the smaller Sim-M dataset (a gain of p < 0.00001).
  • The baseline RoBERTa model without augmentation performed relatively poorly, highlighting the necessity of their proposed method.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Corpus Choice: The choice of augmentation corpora can impact generality. For instance, using humorous restaurant names might be a double-edged sword – helpful for learning variety but potentially unrepresentative. Further experimentation is needed to determine optimal corpora.
  • Dataset Bias: The datasets and their chosen corpora exhibit a US-centric bias for location names. Exploring wider, less US-focused corpora could be beneficial for real-world deployment.
  • Grammatical Differences: While they addressed vocabulary differences, future work could explore targeted data augmentation to address grammatical variations in utterances.

Summary updated

BibTeX
@inproceedings{summerville2020how,
  author = {Adam Summerville and Jordan Hashemi and James Ryan and William Ferguson},
  title = {{How to Tame Your Data: Data Augmentation for Dialog State Tracking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure