TL;DR
This paper proposes a novel approach to cross-domain slot filling by reframing it as a Machine Reading Comprehension (MRC) problem. By transforming slot types into natural language queries, their method leverages large-scale MRC datasets for pre-training, significantly improving performance in low-resource and zero-shot scenarios compared to traditional sequence labeling methods.
What Problem Does This Paper Address?
Task-oriented dialogue systems are increasingly common, and a crucial component is "slot filling" – identifying specific pieces of information (like dates, times, or locations) within a user's request. The challenge arises when these systems need to adapt to new domains with entirely new types of slots, often with very little or no training data. Traditional methods, which treat slot filling as a sequence labeling task (assigning a label to each word), struggle with this "cross-domain" adaptation and data scarcity. They require extensive labeled data for each new domain and slot type, making them slow and expensive to deploy in new scenarios.
Methods
The core innovation of this paper is framing cross-domain slot filling as a Machine Reading Comprehension (MRC) task. Instead of directly labeling tokens, the authors propose:
- Query Generation: Each slot type (e.g., "restaurant_name") is transformed into a natural language question (e.g., "What is the name of the restaurant?"). They explore three strategies for generating these queries:
- Description: Using the slot's textual description.
- Back-translation: Translating descriptions to another language and back to improve naturalness.
- Example: Incorporating actual slot examples into the query.
- MRC Model: A BERT-based MRC model is employed. The user request (context) and the generated slot query are concatenated and fed into BERT. BERT then predicts the start and end positions of the answer span within the context that answers the query.
- Pre-training: To combat data scarcity, the MRC model is pre-trained on a large-scale MRC dataset (SQuAD2.0) before being fine-tuned on the slot-filling task. This allows the model to learn general reading comprehension skills that are transferable to the slot-filling domain.
- Start-End Matching: A novel start-end matching mechanism is introduced to handle cases where a single query might correspond to multiple slot values, prioritizing the most probable and non-overlapping spans.
Datasets
The authors evaluate their approach on two widely used datasets:
- SNIPS: A spoken language understanding dataset with multiple domains and intents. They simulate cross-domain scenarios by treating one domain as the target and the others as sources.
- ATIS: A dataset focused on the airline travel domain, used to test the model's ability to adapt to a completely different domain from SNIPS.
No new benchmarks were introduced; instead, they used existing datasets to demonstrate performance gains in challenging cross-domain settings.
Key Insights and Ideas
- Reframing the Problem: The most significant insight is that MRC is a more natural and effective framework for cross-domain slot filling than traditional sequence labeling, especially when dealing with unseen slots.
- Leveraging Pre-trained MRC Models: Pre-training on large MRC datasets provides a strong foundation, enabling the model to understand natural language queries and extract information even with limited domain-specific training data.
- Query Design Matters: The way slot types are translated into questions significantly impacts performance. Using slot examples in queries proved particularly effective for zero-shot learning.
- Data Efficiency: The MRC approach demonstrates superior data efficiency, achieving strong results with fewer training samples compared to existing methods.
Key Results
The proposed RCSF (Reading Comprehension for Slot Filling) model consistently outperformed state-of-the-art baselines on both SNIPS and ATIS datasets across various settings (zero-shot, 20-shot, 50-shot).
- SNIPS: Achieved average F1 score improvements of 18.37% (zero-shot), 21.28% (20-shot), and 15.43% (50-shot) over the best baseline.
- ATIS: Showed significant gains, including 27.78% improvement in the zero-shot setting.
- Unseen Slots: The model demonstrated a strong ability to recognize unseen slots, outperforming baselines that struggled to adapt. For instance, it could distinguish "owner of the playlist" from "playlist" when "playlist_owner" was an unseen slot.
- Ablation Studies: Pre-training BERT on MRC data was crucial, yielding substantial performance gains. Even without pre-training, the MRC framework was more data-efficient than sequence labeling.
Limitations and Future Work
- Query Generation Nuances: While "Example" queries were generally best, the "Back-translation" method sometimes showed inconsistent performance, suggesting potential information loss during the translation process.
- Future Directions: The authors suggest exploring how to jointly address intent detection and slot filling within a unified MRC framework for cross-domain scenarios.
Summary updated