When a machine learning model learns a sequence of tasks, it typically suffers from catastrophic forgetting—overwriting past knowledge to accommodate new data. In the image domain, standard continual learning strategies mitigate this by saving a buffer of past images (experience replay) or penalizing changes to important weights (regularization). However, as Carta et al. demonstrated in a systematic benchmark (Catastrophic Forgetting in Deep Graph Networks: A Graph Classification Benchmark), these standard "flat" approaches fail when applied to Graph Neural Networks (GNNs).
The failure stems from a fundamental difference in data geometry. In an image, pixels sit on a fixed grid. In a graph, a node's identity and meaning are defined by its relationships with its neighbors. GNNs capture this through message passing, where node representations are iteratively aggregated across edges. If you store isolated nodes for replay, or regularize weights without considering how they affect message propagation, the model loses the structural context—the receptive field—needed to reconstruct past knowledge.
To solve this, a prominent research direction has emerged: Topology-Preserving Continual Graph Learning. The core intuition is that to prevent forgetting in GNNs, models must explicitly protect and replay the relational skeleton of the graph, rather than just individual node features or raw task outputs.
Topology-Aware Regularization: Protecting the Propagation Pathway
One way to prevent forgetting is to restrict changes to the model parameters that matter most for past tasks. In graphs, the most critical parameters are those that govern how information flows across neighborhoods.
Liu et al. introduced this concept with Topology-aware Weight Preserving (TWP) in Overcoming Catastrophic Forgetting in Graph Neural Networks. Instead of calculating parameter importance based solely on the final task loss, TWP explicitly measures how sensitive the local topological structures and neighborhood aggregation strategies are to changes in the weights. By penalizing changes to parameters that significantly alter edge-based information propagation, the model locks in its structural understanding.
As surveyed by Febrinanto et al. in Graph Lifelong Learning: A Survey, this topology-aware regularization ensures that incremental updates do not overwrite the structural integrity of existing relationships. This insight is reinforced by Zhang et al. in CGLB: Benchmark Tasks for Continual Graph Learning, who showed that protecting the topological importance of key nodes and edges stabilizes the model's understanding of global topology far better than standard weight regularization.
Structural Rehearsal: Replaying Subgraphs and Coarsened Motifs
When memory buffers are available, replaying raw, disconnected data points is insufficient because message passing requires edges. Therefore, effective rehearsal requires saving or reconstructing structural sub-units.
Isolated Replay (Fails): [Node A] [Node B] [Node C] --> No structural context
Structural Rehearsal: [Node A] ——— [Node B] ——— [Node C] --> Preserves message-passing pathways
For evolving or temporal networks, Feng et al. argued in Towards Open Temporal Graph Neural Networks that the network's fundamental evolutionary patterns are stored in its local geometry. By replaying fundamental three-node motifs (triads) rather than isolated nodes, the model retains the precise rules of how connections formed in older data distributions.
When dealing with large-scale graphs, storing entire subgraphs becomes memory-prohibitive. To circumvent this, researchers use graph compression techniques that preserve connectivity:
- Graph Coarsening: Han et al. proposed TACO in A Topology-aware Graph Coarsening Framework for Continual Graph Learning. TACO compresses historical graphs into a significantly smaller, coarsened representation that mathematically preserves the original graph's spectral and structural topology. This ensures that the GNN's receptive field remains intact during replay. Similarly, Wan et al. introduced MOTION (MOTION: Multi-Sculpt Evolutionary Coarsening for Federated Continual Graph Learning), demonstrating that preserving the relational skeleton via coarsening is far more critical for long-term memory consolidation than retaining raw node attributes.
- Structural Selection: Instead of random sampling, Xie et al. developed SES (Structural-Entropy-Based Sample Selection for Efficient and Effective Learning), which uses structural entropy decomposition to select a sparse set of replay nodes that best preserve the global organization and relational context of previous tasks.
- Structure Learning: Rather than just saving static edges, Choi et al. showed in DSLR: Diversity Enhancement and Structure Learning for Rehearsal-based Graph Continual Learning that incorporating Graph Structure Learning (GSL) allows the model to dynamically optimize and reconstruct informative neighbor connections for the replayed nodes, drastically reducing the required size of the replay buffer.
Global Alignment and Distillation: Preventing Structural Drift
Beyond local neighborhoods, continual learning causes "structural drift"—a phenomenon where the global coordinate space or relative distances between distant node representations distort over time.
To combat global drift, Hoang et al. developed Universal Graph Continual Learning (UGCL) in Universal Graph Continual Learning. The authors combined experience replay with explicit local and global structure distillation. By forcing the current model's layer-wise representations to maintain the same relative topological distances as the historical model, they achieved a synergistic effect that outperforms data rehearsal alone.
Taking global alignment further, Zhang et al. introduced SAOT (SAOT: Self-Supervised Continual Graph Learning with Structure-Aware Optimal Transport). SAOT utilizes Optimal Transport theory to mathematically align the global topology of the graph with its representation space, ensuring that the macro-level map of node relationships remains stable across long sequences of tasks.
Even when the network structure explicitly changes via edge deletions, the structural history must be accounted for. Li et al. demonstrated in Joint Learning of Evolving Links for Dynamic Network Embedding that maintaining a dual-graph structural memory—one tracking active connections and another preserving deleted links—prevents the loss of historical context and stabilizes node representations over time.
Summary
Topology-preserving continual graph learning works because it aligns with the core mechanism of graph neural networks. GNNs understand the world through connectivity. By designing regularization terms that protect propagation weights, compressing graphs into topology-preserving coarsened structures, and using distillation to prevent global structural drift, these methods ensure that the model's structural memory remains intact across a continuous stream of information.
Key Foundational Work
Overcoming Catastrophic Forgetting in Graph Neural Networks (2021) — contributed 2 concept elements to this direction.
Joint Learning of Evolving Links for Dynamic Network Embedding (2018) — contributed 1 concept elements to this direction.