Kronecker Factorization for Preventing Catastrophic Forgetting in Large-scale Medical Entity Linking

Denis Jered McInerney, Luyang Kong, Kristjan Arumae, Byron C. Wallace, Parminder BhatiaarXiv 2021 · arXiv

Introduction

In the rapidly evolving medical field, AI models must adapt to new diseases, updated ontologies, and changing clinical practices. Ideally, models would be retrained on all historical and new data simultaneously whenever updates occur. However, "joint retraining" is often impractical due to data privacy regulations, expiring contracts, or the prohibitive computational costs of retraining massive architectures like BERT from scratch. This necessitates Continuous Learning, where a model learns tasks sequentially.

The primary hurdle in this paradigm is Catastrophic Forgetting (CF): as a neural network learns a new task, it often overwrites the weights essential for previous tasks, leading to a sharp decline in original performance.

Beyond the Diagonal Assumption

A common technique to mitigate CF is Elastic Weight Consolidation (EWC). EWC identifies "important" weights for a task using the Fisher Information Matrix (FIM) and penalizes changes to these weights during subsequent training. However, standard EWC typically employs a "diagonal" assumption, treating every model parameter as independent. In complex, large-scale models, this simplification fails to capture the intricate dependencies between weights.

The authors propose using Kronecker Factorization (KFC) to relax this assumption. By approximating the FIM as a Kronecker product of two smaller matrices, KFC captures correlations between weights within the same layer. This approach allows the model to find a more nuanced "compromise" between old and new tasks.

Conceptual comparison of Diagonal EWC vs. KFC in approximating the low-error region.

As illustrated in Figure 1, the region of low error for a task is often a "tilted" ellipse rather than a circle aligned with the axes. While diagonal EWC cannot account for this tilt, KFC models these correlations, allowing the optimizer to move along the direction of least interference for the previous task.

Experimental Framework

The researchers evaluated KFC on Medical Entity Linking (MEL), the task of mapping clinical text mentions to standardized medical codes. They utilized three major datasets:

  • MedMention: PubMed abstracts linked to UMLS.
  • MedNorm: Clinical descriptions mapped to MedDRA and SNOMED-CT.
  • 3DNotes: Dictated doctor’s notes linked to ICD-10 codes.

The method was tested across both CNN and Transformer (BERT) architectures, marking the first time Kronecker Factorization has been scaled to such large NLP models.

Key Results

KFC demonstrated a significant advantage over both unregularized baselines and standard diagonal EWC. On BERT-based models, KFC reduced catastrophic forgetting by an average of 51%, nearly doubling the 27% reduction achieved by diagonal EWC.

Performance comparison of KFC against diagonal EWC and no regularization across sequential medical tasks.

As shown in Figure 6, KFC consistently preserved Task A accuracy at much higher levels than diagonal EWC across various task permutations. For instance, in the MedM → 3DNotes sequence using BERT, KFC maintained Task A accuracy at 80.64%, compared to 76.41% for diagonal EWC and 63.77% with no regularization.

Conclusion

By capturing intra-layer weight correlations, Kronecker Factorization provides a scalable and highly effective solution for continuous learning in medical AI. While it introduces a risk of "intransigence"—where the model becomes too resistant to learning new information—the empirical gains in preserving historical knowledge are substantial. Future research will focus on automating hyperparameter selection and exploring inter-layer dependencies to further refine this approach.

Summary updated

BibTeX
@misc{mcinerney2021kronecker,
  author = {Denis Jered McInerney and Luyang Kong and Kristjan Arumae and Byron C. Wallace and Parminder Bhatia},
  title = {{Kronecker Factorization for Preventing Catastrophic Forgetting in Large-scale Medical Entity Linking}},
  year = {2021},
  eprint = {2111.06012},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.06012},
}
Figure