Abstract
Catastrophic forgetting refers to the tendency that a neural network "forgets" the previous learned knowledge upon learning new tasks. Prior methods have been focused on overcoming this problem on convolutional neural networks (CNNs), where the input samples like images lie in a grid domain, but have largely overlooked graph neural networks (GNNs) that handle non-grid data. In this paper, we propose a novel scheme dedicated to overcoming catastrophic forgetting problem and hence strengthen continual learning in GNNs. At the heart of our approach is a generic module, termed as topology-aware weight preserving (TWP), applicable to arbitrary form of GNNs in a plug-and-play fashion. Unlike the main stream of CNN-based continual learning methods that rely on solely slowing down the updates of parameters important to the downstream task, TWP explicitly explores the local structures of the input graph, and attempts to stabilize the parameters playing pivotal roles in the topological aggregation. We evaluate TWP on different GNN backbones over several datasets, and demonstrate that it yields performances superior to the state of the art.
Introduction
Deep learning models are notoriously prone to catastrophic forgetting: as they learn new tasks, they overwrite the weights necessary for previous ones, leading to a sharp decline in performance on older data. While this phenomenon is well-documented in Convolutional Neural Networks (CNNs) for image processing, it remains under-explored in the realm of Graph Neural Networks (GNNs).
GNNs present a unique challenge for continual learning. Unlike images, where data points are independent grid samples, graph data is defined by its topology—the relationships and connections between nodes. Standard continual learning techniques often treat nodes as independent entities, ignoring how information propagates through the graph structure. This oversight leads to a loss of structural knowledge, where the model "forgets" not just the labels, but the optimal way to navigate and aggregate information from the graph.
Methodology: Topology-aware Weight Preserving (TWP)
To address this, we introduce TWP (Topology-aware Weight Preserving), a plug-and-play module designed to stabilize parameters critical to both task performance and graph structure. TWP calculates an importance score for every network parameter, penalizing significant changes to high-importance weights during subsequent task training.
As shown in the framework above, TWP determines parameter importance through two primary components:
- Minimized Loss Preserving: This component identifies parameters essential for minimizing the specific task loss (e.g., node classification). This is a standard approach in continual learning, ensuring the model retains the logic needed for prediction.
- Topological Structure Preserving: This is our core innovation. It identifies parameters pivotal for the aggregation process—how a node gathers data from its neighbors. For Graph Attention Networks (GATs), we monitor attention coefficients. For general GNNs (like GCNs or GINs) that lack native attention, we introduce a temporary "non-parametric attention mechanism" solely to calculate structural importance without altering the model's standard computation.
To maintain model "plasticity" (the ability to learn new tasks), we apply sparse regularization ($l_1$ norm) to these importance scores. This ensures the model only protects the most vital weights, leaving others flexible for future learning.
Experimental Evaluation
We evaluated TWP across diverse benchmarks, including transductive node classification (Corafull, Amazon Computers), inductive node classification (PPI, Reddit), and graph classification (Tox21). We compared TWP against state-of-the-art baselines such as EWC, MAS, and GEM across various GNN backbones (GAT, GCN, GIN).
Key Results
TWP consistently outperformed all baselines in both accuracy and the prevention of forgetting. For instance, on the Corafull dataset with a GAT backbone, TWP achieved an Average Performance (AP) of 89.0%, significantly higher than EWC (86.9%) and MAS (84.1%).
Beyond raw accuracy, TWP demonstrated superior stability. On Amazon Computers, TWP's "Average Forgetting" (AF) score was a mere 0.6%, whereas the standard fine-tuning approach (without protection) suffered a 12.3% loss of previous knowledge. The results across various metrics (Accuracy and F1 score) confirm that preserving topological knowledge is essential for robust continual learning in graphs.
Conclusion
TWP offers a generic, effective solution for overcoming catastrophic forgetting in GNNs by explicitly protecting the parameters that govern graph topology. By balancing task-specific importance with structural preservation, TWP achieves state-of-the-art performance across a wide range of graph-based tasks. Future work will explore more efficient importance estimation and graph-aware data rehearsal strategies to further enhance GNN plasticity.
Summary updated