Introduction
Dialogue State Tracking (DST) is a critical component of modern virtual assistants, tasked with maintaining a representation of the user's goals throughout a multi-turn conversation. Traditional DST systems often rely on fixed ontologies, making them difficult to scale to thousands of services. The SPPD system, developed for the DSTC8 challenge, addresses these limitations through a zero-shot approach that leverages natural language schema descriptions.
Problem Statement: Scalability and Generalization
Existing DST models face two primary hurdles:
- Ontology Explosion: Hard-coding slot names and values for a vast array of third-party services is impractical.
- Zero-Shot Learning: Systems must generalize to new services (e.g., "Pet Grooming") using only semantic cues from slot names and descriptions.
Additionally, models must handle long-range dependencies and "slot carryover," where information from one domain (e.g., a restaurant name) is transferred to another (e.g., a taxi destination).
Methodology: Modular BERT Architecture
The SPPD system utilizes a modular framework consisting of five BERT-based Natural Language Understanding (NLU) models and a rule-based DST module.
Zero-Shot NLU
Rather than using arbitrary IDs, SPPD inputs natural language descriptions of slots and intents from the service schema into BERT. This allows the model to utilize semantic similarity for unseen domains. The five specialized models predict:
- Active Intent: The user's current objective.
- Requested Slots: Information the user seeks.
- Slot Status: Updates to slot values (e.g., "active," "ignored," or "don't care").
- Slot Value/Tagging: Extraction of specific values from the dialogue text.
Previous Intent Dropout
To prevent the model from over-relying on potentially erroneous predictions from previous turns, the authors implemented a dropout mechanism. By randomly omitting the previous intent during training, the model is forced to prioritize the current user utterance, reducing cascading errors. Experiments showed that a high dropout rate (up to 0.9) actually improved intent accuracy by forcing the model to focus on the immediate context.
Value Retrieval and Slot Carryover
To address long-range dependencies, the system employs a Value Retrieval strategy, searching the entire dialogue history if a slot is active but no value is present in the current turn. For Slot Carryover, a binary classifier detects semantic relationships between slots across different domains, enabling seamless value transfer.
Results and Evaluation
The system was evaluated on the Schema Guided Dialogue (SGD) dataset, featuring over 16,000 dialogues across 16 domains. A key metric was performance on "unseen" services to test zero-shot generalization.
As shown in the results table, the SPPD system significantly outperforms the baseline. The inclusion of Value Retrieval (VR) and Slot Carryover (SCO) led to a massive jump in Joint Goal Accuracy (JGA), from 41.9% to 82.1%. The final ensemble reached 90.0% JGA, with a remarkably small performance gap between seen and unseen services (91.6% vs 87.8%), validating the zero-shot approach.
Conclusion
The SPPD system demonstrates that natural language schema descriptions are powerful features for zero-shot DST. By combining modular BERT architectures with robust history-handling strategies like Value Retrieval and Intent Dropout, the system achieves state-of-the-art performance on the SGD dataset. Future work will focus on optimizing these models for production environments to reduce latency while maintaining high accuracy.
Summary updated