Similarity-based Multi-Domain Dialogue State Tracking with Copy Mechanisms for Task-based Virtual Personal Assistants

Jarana Manotumruksa, Jeffrey Dalton, Edgar Meij, Emine YilmazWWW 2022 · OpenReview

Abstract

Task-based Virtual Personal Assistants (VPAs) rely on multi-domain Dialogue State Tracking (DST) models to monitor goals throughout a conversation. Previously proposed models show promising results on established benchmarks, but they have difficulty adapting to unseen domains due to domain-specific parameters in their model architectures. We propose a new Similarity-based Multi-domain Dialogue State Tracking model (SM-DST) that uses retrieval-inspired and fine-grained contextual token-level similarity approaches to efficiently and effectively track dialogue state. The key difference with state-of-the-art DST models is that SM-DST has a single model with shared parameters across domains and slots. Because we base SM-DST on similarity it allows the transfer of tracking information between semantically related domains as well as to unseen domains without retraining. Furthermore, we leverage copy mechanisms that consider the system’s response and the dialogue state from previous turn predictions, allowing it to more effectively track dialogue state for complex conversations. We evaluate SM-DST on three variants of the MultiWOZ DST benchmark datasets. The results demonstrate that SM-DST significantly and consistently outperforms state-of-the-art models across all datasets by absolute 5-18% and 3-25% in the few- and zero-shot settings, respectively.

Introduction

Virtual Personal Assistants (VPAs) must maintain a structured record of user requirements, known as the "dialogue state," across multi-turn, multi-domain conversations. For instance, a user might specify a hotel area and then request a taxi to a specific destination. Tracking these states accurately is difficult due to three primary factors:

  1. Scalability: Many existing models rely on domain-specific parameters, causing model size to explode as new domains (e.g., hotels, taxis, restaurants) are added.
  2. Generalization: Traditional models struggle with "zero-shot" scenarios, where they must handle a domain not encountered during training.
  3. Coreference and Implicit Values: Users frequently refer back to previous entities or accept system suggestions, requiring the model to "copy" values from memory rather than simply extracting them from the current utterance.
Figure 1: Examples of dialogue turns requiring different value extraction strategies, including span-based, inform-based, and refer-based tracking.

Methods: The SM-DST Architecture

To address these challenges, the authors propose SM-DST (Similarity-based Multi-domain Dialogue State Tracking). Unlike previous architectures that utilize separate predictors for every slot, SM-DST employs a unified architecture with shared parameters across all domains and slots.

Figure 2: Comparison between the general architecture of existing DST models (left) and the proposed SM-DST architecture (right), highlighting the shift to shared parameters and similarity-based prediction.

Key Technical Components

  • Dual Encoders: SM-DST uses two BERT-based encoders—one for dialogue context and one for slot descriptions. This enables the model to understand the semantic meaning of a slot (e.g., "destination") rather than treating it as an arbitrary label.
  • Token-Level Correlation Estimator: Rather than comparing coarse sentence-level embeddings, the model calculates similarity between individual tokens in the dialogue and tokens in the slot description. This fine-grained matching is crucial for identifying specific values accurately.
  • Triple Copy Mechanism: To handle complex context, the model integrates three specialized predictors:
  • Slot Operation Predictor: Determines if a slot is mentioned and selects the appropriate handling strategy (e.g., extract new span, refer to previous turn).
  • Span-based Predictor: Extracts values directly from the current dialogue text.
  • Refer-based Predictor: Copies values from the "System Inform Memory" (recent suggestions) or the "Dialogue State Memory" (previous decisions).

Evaluation and Results

The model was evaluated on three versions of the MultiWOZ benchmark (2.1, 2.2, and 2.3), with a specific focus on few-shot (1% training data) and zero-shot performance.

Key Findings

  • Superior Generalization: By basing the model on similarity, knowledge transfers naturally between related domains. For example, learning to track a "destination" for a train allows the model to immediately apply that logic to taxi bookings.
  • Performance Gains: SM-DST consistently outperformed state-of-the-art models like TripPy.
  • Supervised Learning: Achieved a Joint Goal Accuracy (JGA) of 56.86% on MultiWOZ 2.1.
  • Few-Shot: Outperformed baselines by 5-18% absolute. In some domains, SM-DST reached nearly 90% of full-dataset performance with only 20 training dialogues.
  • Zero-Shot: In the "Train" domain, SM-DST achieved a 25% absolute improvement over existing models by leveraging its understanding of similar slots in the "Taxi" domain.

Conclusion

SM-DST demonstrates that treating dialogue state tracking as a token-level similarity task, rather than a classification problem, significantly improves scalability and generalization. By using shared parameters and a robust copy mechanism, the model effectively handles the complexities of multi-domain conversations even with minimal training data. Future work will explore handling purely implicit values and extending the framework to more diverse, open-vocabulary tasks.

Summary updated

Abstract

Task-based Virtual Personal Assistants (VPAs) rely on multi-domain Dialogue State Tracking (DST) models to monitor goals throughout a conversation. Previously proposed models show promising results on established benchmarks, but they have difficulty adapting to unseen domains due to domain-specific parameters in their model architectures. We propose a new Similarity-based Multi-domain Dialogue State Tracking model (SM-DST) that uses retrieval-inspired and fine-grained contextual token-level similarity approaches to efficiently and effectively track dialogue state. The key difference with state-of-the-art DST models is that SM-DST has a single model with shared parameters across domains and slots. Because we base SM-DST on similarity it allows the transfer of tracking information between semantically related domains as well as to unseen domains without retraining. Furthermore, we leverage copy mechanisms that consider the system’s response and the dialogue state from previous turn predictions, allowing it to more effectively track dialogue state for complex conversations. We evaluate SM-DST on three variants of the MultiWOZ DST benchmark datasets. The results demonstrate that SM-DST significantly and consistently outperforms state-of-the-art models across all datasets by absolute 5-18% and 3-25% in the few- and zero-shot settings, respectively.
BibTeX
@inproceedings{manotumruksa2022similarity,
  author = {Jarana Manotumruksa and Jeffrey Dalton and Edgar Meij and Emine Yilmaz},
  title = {{Similarity-based Multi-Domain Dialogue State Tracking with Copy Mechanisms for Task-based Virtual Personal Assistants}},
  booktitle = {The Web Conference},
  year = {2022},
  url = {https://openreview.net/forum?id=w1BdyaTiIh},
}
Figure