A Progressive Model to Enable Continual Learning for Semantic Slot Filling

Yilin Shen, Xiangyu Zeng, Hongxia JinEMNLP 2019 · OpenReview

Introduction

In Spoken Language Understanding (SLU), semantic slot filling is the critical task of identifying key information within a sentence—for instance, identifying "London" as the destination_city in the query "book a flight to London." While traditional models are trained "offline" on static datasets, real-world voice assistants must adapt to new slang, personalized expressions, and evolving topics post-deployment.

Current approaches face a "triple threat": Catastrophic Forgetting, where fine-tuning on new data erases old knowledge; Data Privacy & Scale, where storing all historical data for retraining is unfeasible; and Efficiency, as retraining massive networks for every update is too slow.

ProgModel: A Progressive Architecture

To address these challenges, the authors propose ProgModel, an additive learning architecture. Instead of modifying a static base model ($M_0$), the system appends a small "Expanded Component" ($M_t$) for each new batch of data. The original components are frozen, physically preserving their knowledge.

ProgModel Architecture: The system expands by adding new components while freezing old ones, utilizing a context gate for knowledge transfer.

The "secret sauce" of this architecture is the Context Gate. This mechanism creates lateral connections between existing components and the new one. A sigmoid-based gate determines how much information to transfer from the previously learned context, allowing the new module to "stand on the shoulders" of its predecessors. This lateral transfer ensures the model doesn't have to re-learn basic grammar but can focus on new expressions.

For inference, the authors introduced two strategies within an Inference Decision Engine (IDE):

  • t-ProgModel: Utilizes only the latest component for decision making.
  • c-ProgModel: A combined approach that evaluates outputs from all components, selecting the label with the highest probability while filtering by component-specific vocabulary.

Experimental Evaluation

The model was evaluated on two gold-standard SLU benchmarks: ATIS (Airline Travel Information Systems) and Snips (covering seven diverse domains like "Play Music" and "Book Restaurant"). To simulate real-world continual learning, utterances were grouped into batches to mimic the arrival of new expression clusters over time.

Key Results

ProgModel consistently outperformed standard fine-tuning competitors, achieving up to a 4.24% boost on ATIS and 3.03% on Snips. Notably, the model performed nearly as well as an "upper bound" baseline—a model retrained on the entire cumulative dataset—despite never having access to old data during new training phases.

Performance Comparison: ProgModel variants (t-ProgModel and c-ProgModel) consistently outperform fine-tuning approaches across various domains and batches.

Beyond accuracy, ProgModel offers significant efficiency gains. New components typically converge in just 10 epochs, compared to the 100 epochs required for full retraining. Furthermore, the model manages its memory footprint by only expanding the embedding layer for truly novel words. While the model grows, it does so much more slowly than naive solutions; on ATIS, ProgModel reached 33MB after four batches, whereas a "copy-and-fine-tune" approach ballooned to 95MB.

Discussion and Future Work

By freezing old weights and adding new ones, ProgModel bypasses catastrophic forgetting entirely. However, some limitations remain. The model size increases linearly with the number of training batches, which might eventually necessitate periodic "re-initialization" of a new base model in production. Additionally, as components accumulate, the Inference Decision Engine's complexity grows, potentially impacting real-time latency. Future research may explore "true" online learning from single utterances rather than batches.

Summary updated

BibTeX
@inproceedings{shen2019progressive,
  author = {Yilin Shen and Xiangyu Zeng and Hongxia Jin},
  title = {{A Progressive Model to Enable Continual Learning for Semantic Slot Filling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=dVQFNXpR7pl},
}
Figure