Keep and Learn: Continual Learning by Constraining the Latent Space for Knowledge Preservation in Neural Networks

Hyo-Eun Kim, Seungwook Kim, Jaehwan LeeMICCAI 2018 · DOI

TL;DR

This paper tackles the "catastrophic forgetting" problem in continual learning, where neural networks forget previously learned information when trained on new data. They propose a novel method that constrains the network's internal "latent space" to remain informative about past data, effectively preserving knowledge without needing to store old datasets. This approach shows improved performance on image classification tasks compared to existing methods.

What Problem Does This Paper Address?

Machine learning models, especially deep neural networks, are data-hungry. However, in many real-world scenarios, access to data is restricted or sequential. For instance, medical data from different hospitals might be processed independently due to privacy concerns. This creates a challenge for continual learning: how can a model learn from new data without forgetting what it learned from previous, now inaccessible, data? This phenomenon is known as "catastrophic forgetting," where the model's performance on older tasks degrades significantly as it learns new ones. Existing methods like fine-tuning, Learning without Forgetting (LwF), and Elastic Weight Consolidation (EWC) attempt to mitigate this, but they often have limitations. LwF relies on pseudo-labels from previous outputs, while EWC directly constrains model parameters. This paper aims to find a more robust way to preserve knowledge by focusing on the internal representation (latent space) of the data.

Methods

The core idea of this paper is to model the high-level feature space (latent space) and the output space as being mutually informative. They propose a method that, during the initial training stage, learns to make the latent representation informative about both the input data and the task's output. This is achieved by jointly optimizing for the standard task-specific loss (e.g., classification) and a reconstruction loss. The reconstruction loss encourages the latent representation to be reconstructible, ensuring it retains meaningful information about the input.

In subsequent learning stages, the parameters responsible for generating this informative latent space are frozen. The model then learns new tasks by adapting its output layer while ensuring that the new latent representations are still constrained to lie within the space modeled in the first stage. This is enforced by using a modified LwF loss that leverages pseudo-labels derived from the initial stage's output, effectively guiding the new representations back towards the preserved knowledge. Essentially, they are creating a "template" for the latent space in the first stage and forcing subsequent learning to conform to it.

Datasets

The proposed method was evaluated on the following datasets:

  • CIFAR-10/100: Standard image classification benchmarks with 10 and 100 classes, respectively.
  • Chest X-rays (CXRs): A real-world medical dataset for tuberculosis classification. This dataset is notable as it represents a practical application where data privacy and sequential processing are critical.

Key Insights and Ideas

  • Latent Space as a Knowledge Anchor: The paper highlights that the latent space, not just the output logits or model weights, can serve as a crucial anchor for preserving knowledge. By ensuring this space remains informative, the model can better retain past learning.
  • Mutual Informativeness: The concept of making the latent space and output mutually informative is key. This means the latent representation should contain enough information to predict the output, and vice-versa, fostering a richer and more robust representation.
  • First Stage as a Knowledge Blueprint: The initial training stage is critical for defining the "ideal" latent space. Freezing parts of the model that generate this space in later stages effectively constrains new learning to stay within this established knowledge boundary.
  • Reconstruction Loss for Information Preservation: The use of a reconstruction loss is a clever way to ensure the latent space doesn't become trivial (i.e., losing information) while learning the primary task.

Key Results

  • Improved Performance: The proposed method consistently outperformed baseline methods (Fine-tuning, EWC, LwF, LwF+, EWCLWF, EWCLWF+) in terms of error rates on CIFAR-10/100 and AUC on the CXR dataset across multiple learning stages.
  • Knowledge Preservation: For CIFAR-10, the proposed method preserved 89.40% of the first stage's knowledge at stage-4, outperforming other methods. Similar strong preservation was observed for CIFAR-100.
  • Medical Data Efficacy: The method demonstrated its effectiveness on a real-world medical dataset (CXRs), showing its practical applicability beyond standard benchmarks.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the proposed method involves an additional reconstruction loss and potentially more complex model architecture (e.g., modified ResNet), which might increase training time and computational resources.
  • Generalization to Other Tasks: The paper primarily focuses on image classification. The authors suggest that further experimental analysis is needed for tasks like lesion detection or segmentation, indicating this as a direction for future work.
  • Tuning Hyperparameters: The effectiveness of the method relies on carefully chosen weighting constants for the reconstruction and LwF losses (λrec, λLwF), which might require extensive hyperparameter tuning.

Summary updated

BibTeX
@inproceedings{kim2018keep,
  author = {Hyo-Eun Kim and Seungwook Kim and Jaehwan Lee},
  title = {{Keep and Learn: Continual Learning by Constraining the Latent Space for Knowledge Preservation in Neural Networks}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2018},
  doi = {10.1007/978-3-030-00928-1\_59},
}
Figure