Introduction
In industrial machine learning, particularly for Spoken Language Understanding (SLU) systems like Amazon Alexa, the constant rollout of new features—such as new intents and slots—presents a significant scaling challenge. Traditionally, models are retrained from scratch on the entire dataset whenever new features are added. This process is computationally expensive and slow. Researchers at Amazon Alexa AI propose an alternative: Local-to-Global Learning (LGL), an iterative training schedule that reduces training time while improving model accuracy.
The Challenge of Iterative Training
Production environments frequently introduce new capabilities (e.g., adding "Find Soundtrack" to a Music domain that already supports "Play Music"). While "warm-starting" a model with previous weights is a common technique to save time, it often leads to catastrophic forgetting or poor generalization in production SLU models. Consequently, the industry standard remains retraining from scratch, which is environmentally and financially taxing.
Methodology: Local-to-Global Learning (LGL)
The authors adapted Local-to-Global Learning (LGL), a curriculum learning strategy, for SLU. The process involves training the model in stages rather than all at once.
The LGL framework consists of three main phases:
- Local Phase: The model is initially trained on a subset of classes. Data for other classes is masked using a placeholder label.
- Iterative Expansion: New batches of classes are unmasked in subsequent iterations.
- Weight Transfer: Encoder and embedding weights are transferred between iterations, while output layers are re-initialized to accommodate new classes.
The study evaluated two scenarios: "Original LGL," where all data is available but labels are masked, and "Production LGL," which simulates real-world data arrival cycles.
Key Insights and Results
The research, conducted on a large-scale German voice assistant dataset focusing on the Music domain, revealed several critical findings:
- Curriculum Benefits: By starting with simpler tasks, the model develops more robust encoder representations.
- Optimal Batching: The number of batches used to introduce classes is a critical hyperparameter. The researchers found a "sweet spot" between 3 and 5 batches.
As shown in the results, splitting the data into too many batches (e.g., 10) leads to diminishing returns or overfitting, as the model spends excessive time on small data subsets.
- Performance Gains: The best LGL configuration improved the Intent Classification Error Rate (ICER) by 9.1% and the Semantic Error Rate (SEMER) by 7.3% compared to the from-scratch baseline.
- Efficiency: LGL achieved up to a 25% reduction in training time for individual iterations, demonstrating that warm-starting is viable when combined with a structured learning schedule.
Conclusion
Local-to-Global Learning provides a robust framework for the continuous expansion of production SLU models. By treating feature addition as a curriculum learning problem, LGL avoids the pitfalls of catastrophic forgetting while significantly reducing the computational overhead of model maintenance. Future work will focus on automating hyperparameter selection and testing the approach across more diverse domains.
Summary updated