Abstract
Building conversational AI for new domains is traditionally expensive, requiring thousands of hand-labeled examples for each new topic. This paper introduces Bag of Experts (BoE), an architecture that reuses pre-trained models for common concepts like "dates" or "locations" to boost the performance of new models. This approach is particularly effective when training data is scarce, allowing for significant improvements in slot tagging accuracy without the overhead of retraining on massive pooled datasets.
Introduction
Personal digital assistants rely on slot tagging to extract meaning from user utterances. For example, in the phrase "Book a room in Austin for next Tuesday," the system must identify "Austin" as a location and "next Tuesday" as a date.
The primary challenge in scaling these systems is the "cold start" problem: every new domain (e.g., "Shopping" or "Real Estate") typically requires a brand-new annotated dataset. While many domains share common concepts, traditional "data-driven adaptation"—pooling all raw data—leads to exploding training times. The authors propose a more efficient alternative: reusing the intelligence of existing models through a model-driven approach.
Methods: Model-Driven Adaptation
The proposed Bag of Experts (BoE) architecture shifts from sharing raw data to sharing the outputs of specialized "expert" models.
- Expert Models: Specialized models are trained on large datasets for universal concepts, such as a "Timex" expert for dates and a "Location" expert.
- LSTM-BoE: In a standard Long Short-Term Memory (LSTM) network, word embeddings are augmented by "consulting" the experts. The expert's output for a word is concatenated with standard features before processing by the main network.
- CRF-BoE: Similarly, in Conditional Random Fields (CRF), expert predictions serve as additional input features for the target domain classifier.
The "Bag" architecture allows multiple experts to be plugged in simultaneously, providing a modular way to transfer knowledge to new domains.
Datasets and Experimental Setup
The study utilized data from Microsoft Cortana across 10 distinct target domains, including Fashion, Flight Status, and Travel. To simulate real-world constraints, the authors tested the models using "low-data" subsets of 500, 1000, and 2000 training instances per domain. Two primary experts were used: Timex and Location, each trained on approximately 1 million utterances.
Key Results
The BoE models consistently outperformed standard baselines, with the most significant gains observed in the most data-constrained scenarios.
- Significant F1 Gains: With only 500 training instances, the LSTM-BoE model achieved a staggering 12.16% average improvement in F1-score over the baseline.
- Data Efficiency: An LSTM-BoE model trained on just 500 instances outperformed a standard CRF model trained on 2000 instances. This suggests that the BoE architecture can reduce manual data annotation requirements by up to 75% while maintaining high quality.
- Scalability: Unlike data pooling, BoE keeps training times proportional to the new domain's data size, facilitating faster iteration.
Discussion and Limitations
While BoE provides a powerful solution for the "cold start" problem, the authors noted diminishing returns as domain-specific data increases. Once a target model has sufficient data to learn common concepts itself, the relative benefit of the expert decreases. Future research is needed to automate expert selection for unknown domains and to identify the precise "break-even point" where the computational cost of an expert outweighs its performance benefit.
Summary updated