Self-Organizing Incremental Neural Networks for Continual Learning

Chayut Wiwatcharakoses, Daniel BerrarIJCAI 2019 · DOI

TL;DR

This paper introduces SOINN+, a novel self-organizing incremental neural network designed for continual learning. It tackles the critical challenge of "catastrophic forgetting" by making node deletion an intrinsic part of the learning process, allowing the network to gracefully adapt to new information without losing previously acquired knowledge.

What Problem Does This Paper Address?

The paper addresses the fundamental challenge in machine learning known as continual learning (or lifelong learning). Imagine a machine learning model that, after learning to recognize cats, completely forgets how to recognize dogs when it's subsequently trained on dog images. This is the problem of catastrophic forgetting: when a model learns new information, it often overwrites or loses previously learned knowledge. This is a significant hurdle for creating AI systems that can learn and adapt incrementally over time, much like humans do. Traditional neural networks are often trained for a single task and struggle to incorporate new data or adapt to changing data distributions without extensive retraining, which is often impractical or impossible with large, streaming datasets.

Methods

The core contribution is the development of SOINN+, an evolution of Self-Organizing Incremental Neural Networks (SOINN). SOINN models learn by creating a topological map of the input data. SOINN+ introduces two key novel concepts to address catastrophic forgetting:

  • Trustworthiness of a node: This metric reflects the system's confidence that a node represents genuine data patterns rather than noise.
  • (Un)utility of a node: This measures how relevant a node is to the current learning task.

Nodes that become untrustworthy or irrelevant are candidates for deletion. This makes node deletion an active, integrated part of the learning process, allowing SOINN+ to "learn to forget" gracefully. The paper also notes that current SOINN methods often delete nodes at fixed, user-defined intervals, which is considered unnatural.

Datasets

The paper uses a synthetic dataset for illustration:

  • 4500 bivariate positive cases (blue dots) with X+, Y+ ~ N(50,10).
  • 4500 bivariate negative cases (green dots) with X_,Y_ ~ N(100, 10).
  • 1000 instances of uniformly distributed noise (red dots).

This dataset is used to demonstrate the topological mapping capabilities of SOINN+.

Key Insights and Ideas

  • "Learning to forget" is crucial for continual learning: The paper argues that gracefully discarding irrelevant or noisy information is as important as acquiring new knowledge for robust lifelong learning.
  • Intrinsic node deletion improves resilience: By making node deletion a dynamic part of the learning process, SOINN+ can adapt to new data and noise more effectively than methods with fixed deletion schedules.
  • Trustworthiness and utility as key metrics: These novel concepts provide a principled way to identify and remove redundant or erroneous nodes, leading to a more compact and accurate representation of the data.
  • Potential for transfer learning: The authors suggest that SOINN+'s ability to adapt without catastrophic forgetting lays the groundwork for future research into enabling transfer learning capabilities.

Key Results

The paper presents a qualitative result:

  • Figure 2 shows a SOINN+ network with 125 nodes after processing 10,000 input cases (including noise). This network is described as a compact representation of the training data.
  • The resulting network has two unconnected nodes identified as noise and candidates for deletion, demonstrating the system's ability to handle noisy data.
  • The authors state that their ongoing research shows SOINN+ can accommodate new classes without catastrophically interfering with existing network structures.

Limitations and Future Work

Limitations: * The paper acknowledges that the Euclidean distance metric used for similarity measurement in the illustrated example is not suitable for high-dimensional data. * The current SOINN methods have an "unnatural network evolution" due to periodic node deletion.

Future Work: * Investigate more suitable distance metrics for high-dimensional data, such as fractional distance. * Extend the network architecture by adding additional layers to learn hierarchical representations, similar to deep convolutional neural networks. * Explore how these extensions can facilitate transfer learning. * Develop a nearest-neighbor classifier that utilizes the learned nodes as candidate prototypes.

Summary updated

BibTeX
@inproceedings{wiwatcharakoses2019self,
  author = {Chayut Wiwatcharakoses and Daniel Berrar},
  title = {{Self-Organizing Incremental Neural Networks for Continual Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/927},
}
Figure