Abstract
Task-based Virtual Personal Assistants (VPAs) rely on multi-domain Dialogue State Tracking (DST) models to monitor goals throughout a conversation. Previously proposed models show promising results on established benchmarks, but they have difficulty adapting to unseen domains due to domain-specific parameters in their model architectures. We propose a new Similarity-based Multi-domain Dialogue State Tracking model (SM-DST) that uses retrieval-inspired and fine-grained contextual token-level similarity approaches to efficiently and effectively track dialogue state. The key difference with state-of-the-art DST models is that SM-DST has a single model with shared parameters across domains and slots. Because we base SM-DST on similarity it allows the transfer of tracking information between semantically related domains as well as to unseen domains without retraining. Furthermore, we leverage copy mechanisms that consider the system’s response and the dialogue state from previous turn predictions, allowing it to more effectively track dialogue state for complex conversations. We evaluate SM-DST on three variants of the MultiWOZ DST benchmark datasets. The results demonstrate that SM-DST significantly and consistently outperforms state-of-the-art models across all datasets by absolute 5-18% and 3-25% in the few- and zero-shot settings, respectively.
Introduction
Virtual Personal Assistants (VPAs) must maintain a structured record of user requirements, known as the "dialogue state," across multi-turn, multi-domain conversations. For instance, a user might specify a hotel area and then request a taxi to a specific destination. Tracking these states accurately is difficult due to three primary factors:
- Scalability: Many existing models rely on domain-specific parameters, causing model size to explode as new domains (e.g., hotels, taxis, restaurants) are added.
- Generalization: Traditional models struggle with "zero-shot" scenarios, where they must handle a domain not encountered during training.
- Coreference and Implicit Values: Users frequently refer back to previous entities or accept system suggestions, requiring the model to "copy" values from memory rather than simply extracting them from the current utterance.
Methods: The SM-DST Architecture
To address these challenges, the authors propose SM-DST (Similarity-based Multi-domain Dialogue State Tracking). Unlike previous architectures that utilize separate predictors for every slot, SM-DST employs a unified architecture with shared parameters across all domains and slots.
Key Technical Components
- Dual Encoders: SM-DST uses two BERT-based encoders—one for dialogue context and one for slot descriptions. This enables the model to understand the semantic meaning of a slot (e.g., "destination") rather than treating it as an arbitrary label.
- Token-Level Correlation Estimator: Rather than comparing coarse sentence-level embeddings, the model calculates similarity between individual tokens in the dialogue and tokens in the slot description. This fine-grained matching is crucial for identifying specific values accurately.
- Triple Copy Mechanism: To handle complex context, the model integrates three specialized predictors:
- Slot Operation Predictor: Determines if a slot is mentioned and selects the appropriate handling strategy (e.g., extract new span, refer to previous turn).
- Span-based Predictor: Extracts values directly from the current dialogue text.
- Refer-based Predictor: Copies values from the "System Inform Memory" (recent suggestions) or the "Dialogue State Memory" (previous decisions).
Evaluation and Results
The model was evaluated on three versions of the MultiWOZ benchmark (2.1, 2.2, and 2.3), with a specific focus on few-shot (1% training data) and zero-shot performance.
Key Findings
- Superior Generalization: By basing the model on similarity, knowledge transfers naturally between related domains. For example, learning to track a "destination" for a train allows the model to immediately apply that logic to taxi bookings.
- Performance Gains: SM-DST consistently outperformed state-of-the-art models like TripPy.
- Supervised Learning: Achieved a Joint Goal Accuracy (JGA) of 56.86% on MultiWOZ 2.1.
- Few-Shot: Outperformed baselines by 5-18% absolute. In some domains, SM-DST reached nearly 90% of full-dataset performance with only 20 training dialogues.
- Zero-Shot: In the "Train" domain, SM-DST achieved a 25% absolute improvement over existing models by leveraging its understanding of similar slots in the "Taxi" domain.
Conclusion
SM-DST demonstrates that treating dialogue state tracking as a token-level similarity task, rather than a classification problem, significantly improves scalability and generalization. By using shared parameters and a robust copy mechanism, the model effectively handles the complexities of multi-domain conversations even with minimal training data. Future work will explore handling purely implicit values and extending the framework to more diverse, open-vocabulary tasks.
Summary updated