Local-to-global learning for iterative training of production SLU models on new features

Yulia Grishina, Daniil SorokinNAACL 2022 · DOI

Introduction

In industrial machine learning, particularly for Spoken Language Understanding (SLU) systems like Amazon Alexa, the constant rollout of new features—such as new intents and slots—presents a significant scaling challenge. Traditionally, models are retrained from scratch on the entire dataset whenever new features are added. This process is computationally expensive and slow. Researchers at Amazon Alexa AI propose an alternative: Local-to-Global Learning (LGL), an iterative training schedule that reduces training time while improving model accuracy.

The Challenge of Iterative Training

Production environments frequently introduce new capabilities (e.g., adding "Find Soundtrack" to a Music domain that already supports "Play Music"). While "warm-starting" a model with previous weights is a common technique to save time, it often leads to catastrophic forgetting or poor generalization in production SLU models. Consequently, the industry standard remains retraining from scratch, which is environmentally and financially taxing.

Methodology: Local-to-Global Learning (LGL)

The authors adapted Local-to-Global Learning (LGL), a curriculum learning strategy, for SLU. The process involves training the model in stages rather than all at once.

Overview of the Local-to-Global Learning (LGL) iterative training process.

The LGL framework consists of three main phases:

  1. Local Phase: The model is initially trained on a subset of classes. Data for other classes is masked using a placeholder label.
  2. Iterative Expansion: New batches of classes are unmasked in subsequent iterations.
  3. Weight Transfer: Encoder and embedding weights are transferred between iterations, while output layers are re-initialized to accommodate new classes.

The study evaluated two scenarios: "Original LGL," where all data is available but labels are masked, and "Production LGL," which simulates real-world data arrival cycles.

Key Insights and Results

The research, conducted on a large-scale German voice assistant dataset focusing on the Music domain, revealed several critical findings:

  • Curriculum Benefits: By starting with simpler tasks, the model develops more robust encoder representations.
  • Optimal Batching: The number of batches used to introduce classes is a critical hyperparameter. The researchers found a "sweet spot" between 3 and 5 batches.
Impact of the number of batches on SEMER and ICER metrics.

As shown in the results, splitting the data into too many batches (e.g., 10) leads to diminishing returns or overfitting, as the model spends excessive time on small data subsets.

  • Performance Gains: The best LGL configuration improved the Intent Classification Error Rate (ICER) by 9.1% and the Semantic Error Rate (SEMER) by 7.3% compared to the from-scratch baseline.
  • Efficiency: LGL achieved up to a 25% reduction in training time for individual iterations, demonstrating that warm-starting is viable when combined with a structured learning schedule.

Conclusion

Local-to-Global Learning provides a robust framework for the continuous expansion of production SLU models. By treating feature addition as a curriculum learning problem, LGL avoids the pitfalls of catastrophic forgetting while significantly reducing the computational overhead of model maintenance. Future work will focus on automating hyperparameter selection and testing the approach across more diverse domains.

Summary updated

BibTeX
@inproceedings{grishina2022local,
  author = {Yulia Grishina and Daniil Sorokin},
  title = {{Local-to-global learning for iterative training of production SLU models on new features}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-industry.13},
}
Figure