Abstract
Task-oriented dialog systems increasingly rely on deep learning-based slot filling models, usually needing extensive labeled training data for target domains. Often, however, little to no target domain training data may be available, or the training and target domain schemas may be misaligned, as is common for web forms on similar websites. Prior zero-shot slot filling models use slot descriptions to learn concepts, but are not robust to misaligned schemas. We propose utilizing both the slot description and a small number of examples of slot values, which may be easily available, to learn semantic representations of slots which are transferable across domains and robust to misaligned schemas. Our approach outperforms state-of-the-art models on two multi-domain datasets, especially in the low-data setting.
Introduction
Scaling task-oriented dialog systems (e.g., Siri, Alexa) to new domains is notoriously difficult when training data is scarce and slot schemas do not match. In these systems, slot filling is the critical task of extracting specific parameters—such as a flight date or cuisine—from a user's utterance. While traditional models require massive labeled datasets, researchers have turned to zero-shot learning, typically by conditioning models on textual slot descriptions (e.g., "departure city").
However, description-based models often fail when domains have misaligned schemas. For instance, the slot name depart might refer to a date on one travel site and a city on another. Relying solely on the slot name leads to negative transfer. This research proposes a novel solution: conditioning the model not just on the slot's name, but on a few examples of what its values look like (e.g., "Boston", "Nov 5").
Methodology
The authors frame slot filling as a conditional sequence tagging task using Inside-Outside-Begin (IOB) tags. The architecture maps three distinct inputs into a shared embedding space using pre-trained word embeddings and character-level CNNs:
- Utterance Encoder: A bidirectional GRU processes the user's sentence.
- Slot Description Encoder: This component mean-pools the embeddings of the slot's name or description.
- Slot Example Encoder: This is the core novelty. It takes $K$ example values for the target slot and uses an attention mechanism to weigh these examples relative to the current word in the user's utterance.
A bidirectional LSTM tagger then concatenates these representations to classify each token. To ensure robustness, the authors employ a training strategy where slot value tokens are randomly replaced with a special <OOV> token 30% of the time, forcing the model to learn from context rather than simple string matching.
Experimental Results
The model was evaluated on two datasets: SNIPS (a public SLU dataset with 39 slots) and XSchema (an in-house dataset designed to simulate conflicting schemas from real-world web forms).
Key Findings
- Disambiguation: Example values successfully resolve schema collisions. Providing "JFK" as an example for a
departslot clearly identifies it as a location rather than a date. - Optimal Example Count: For strictly zero-shot scenarios (unseen slots), providing just 2 example values yields the best results. Increasing the count beyond this can lead to overfitting on exact string matches. In few-shot settings (e.g., 50 training instances), performance continues to scale up to 10 examples.
- Performance Gains: On SNIPS, the model achieved a 40.6% F1 score, outperforming strong zero-shot baselines by ~2.9%. On XSchema, the model showed high robustness, with F1 gains of up to 10% on intents like FindFlights and BookRoom.
Conclusion
This work demonstrates that incorporating a small number of example values significantly enhances the robustness of zero-shot slot filling, particularly against misaligned schemas. While the current per-slot conditioning approach faces some computational efficiency challenges and potential prediction overlaps, it sets a new state-of-the-art for cross-domain transfer. Future work will explore joint modeling of multiple slots and the integration of zero-shot entity recognition to further automate the process.
Summary updated