Alleviating Catastrophic Forgetting with Modularity for Continuously Learning Linked Open Data

Lu Chen, Masayuki MurataInternational Journal of Computer Theory and Engineering 2018 · DOI

Abstract

Nowadays, Linked Open Data is spreading year by year, and its further utilization is expected. Because of the large size of data, Linked Open Data is attempted to learn by using neural networks. Since the data is still scaling in various region, unlike the existing neural network specialized to learn only one region, a neural network which can continuously learn wide region of knowledge is needed. However, neural network is known in its problem, catastrophic forgetting, which is to lose previously acquired skills when learning a new skill. Though existing researches said enhancing modularity can overcome this problem since it can reduce interference between tasks, those researches consider the number of learning tasks is given in advance, and it is not applicable for continuous learning. In this paper, we propose a design approach of neural network reducing modularity expecting that unspecialization can mitigate catastrophic forgetting for continuous learning. Our results show that, although, as we can expect, a neural network with high modularity can mitigate forgetting for tasks learned just before because of the low interference, a neural network with low modularity is better for the worst case when evaluating for all the tasks it learned in the past.

Introduction

As Linked Open Data (LOD) continues to expand, the demand for neural networks capable of continuous learning—acquiring new knowledge without overwriting previous information—has grown. This overwriting phenomenon, known as catastrophic forgetting, remains a significant hurdle. Conventional wisdom suggests that "modularity"—isolating functional units within a network—can mitigate this by reducing task interference. However, most modular approaches assume a fixed number of tasks, which is incompatible with the open-ended nature of LOD. This research investigates whether "unspecialization" through low-modularity structures might offer better long-term robustness.

Methodology

The study compares two distinct neural network topologies:

  1. High Modularity ($H^{high}$): Networks with distinct subsets of nodes and links assigned to specific learning tasks.
  2. Low Modularity ($H^{low}$): Networks where these boundaries are blurred. These were generated by applying Simulated Annealing to high-modularity networks, randomly exchanging nodes and rewiring links until modularity reached near zero.
The complete neural network architecture used for learning Linked Open Data categories.

Tree-Structured Categorization and Evolutionary Learning

To handle the scaling nature of LOD, the authors represent categories as a tree structure. The network learns to map "object nodes" (e.g., specific animals) to "attribute nodes" (e.g., "wings," "carnivore") via intermediate "category nodes." The NEAT (NeuroEvolution of Augmenting Topologies) and HyperNEAT algorithms were employed to evolve weights and connections.

An example of a specialized sub-network (module) activated for a specific learning task in the high-modularity condition.

Experimental Setup

The researchers utilized the BBC Nature Linked Open Data set, focusing on animal attributes. Continuous learning was simulated by punctuating the data into sequential tasks, each consisting of three animal-attribute sets.

Results and Discussion

Performance was evaluated using the F-measure across three temporal scales:

  • Current Task (PT1): High-modularity networks excelled, benefiting from low immediate interference.
  • Recent History (PT5): Performance converged between high and low modularity after approximately 100 generations.
  • All Past Tasks (PTALL): In this "worst-case" scenario, low-modularity networks significantly outperformed high-modularity ones after 400 generations.

While high modularity acts as a "short-term memory" specialist, low modularity functions as a "long-term generalist." The overlapping structure of low-modularity networks allows them to maintain a more stable average performance across the entire history of tasks, despite higher immediate interference.

Conclusion

This paper challenges the assumption that high modularity is always superior for preventing catastrophic forgetting. In continuous learning environments where the task count is unknown, unspecialized, low-modularity networks provide superior retention of historical knowledge. Future work will explore larger task scales and more diverse LOD datasets to further validate these findings.

Summary updated

Abstract

Nowadays, Linked Open Data is spreading year by year, and its further utilization is expected. Because of the large size of data, Linked Open Data is attempted to learn by using neural networks. Since the data is still scaling in various region, unlike the existing neural network specialized to learn only one region, a neural network which can continuously learn wide region of knowledge is needed. However, neural network is known in its problem, catastrophic forgetting, which is to lose previously acquired skills when learning a new skill. Though existing researches said enhancing modularity can overcome this problem since it can reduce interference between tasks, those researches consider the number of learning tasks is given in advance, and it is not applicable for continuous learning. In this paper, we propose a design approach of neural network reducing modularity expecting that unspecialization can mitigate catastrophic forgetting for continuous learning. Our results show that, although, as we can expect, a neural network with high modularity can mitigate forgetting for tasks learned just before because of the low interference, a neural network with low modularity is better for the worst case when evaluating for all the tasks it learned in the past.
BibTeX
@article{chen2018alleviating,
  author = {Lu Chen and Masayuki Murata},
  title = {{Alleviating Catastrophic Forgetting with Modularity for Continuously Learning Linked Open Data}},
  journal = {International Journal of Computer Theory and Engineering},
  year = {2018},
  doi = {10.7763/ijcte.2018.v10.1196},
}
Figure