When a machine learning model learns new information sequentially, it tends to overwrite what it previously mastered—a flaw known as catastrophic forgetting. In standard continual learning settings, data points like images or text documents are independent of one another. If a model learns to classify a new breed of dog, the pixels of previously learned cat images remain unchanged.
Graphs, however, break the assumption of independence. In a graph, data points (nodes) are explicitly bound together by relationships (edges). Graph Neural Networks (GNNs) rely on message passing, where a node’s representation is computed by aggregating features from its structural neighbors. This interconnectedness introduces a unique challenge: when a GNN updates its weights to accommodate a new snapshot of data or an emerging class, those weight changes propagate across the entire topology. As a result, the representations of historical nodes shift drastically, causing severe performance degradation on past data.
As Antonio Carta et al. demonstrate in Catastrophic Forgetting in Deep Graph Networks: A Graph Classification Benchmark, topological structure does not inherently protect a network from forgetting; graph models lose past knowledge just as severely as structure-agnostic models. Managing this requires navigating what Peyman Moghadam et al. term the stability-plasticity dilemma in Inductive Graph Few-shot Class Incremental Learning: the model must remain plastic enough to assimilate novel structural patterns while maintaining the stability required to prevent feature drift in older classes.
To solve this, researchers have developed specialized strategies that preserve structural context, isolate parameters, or align latent geometric spaces over time.
Rehearsal via Graph Condensation and Structure Repair
The most straightforward defense against forgetting is experience replay—re-training the model on a small buffer of historical data alongside the new data stream. However, simply saving a random sample of old nodes fails in graph domains. If you store a node but discard its neighbors, the message-passing mechanism during replay will aggregate an entirely different neighborhood context, distorting the node's representation.
To preserve structural context without incurring massive storage costs, recent frameworks synthesize or optimize the replay memory:
- Graph Condensation: Instead of keeping raw, incomplete samples of old subgraphs, Ruihong Qiu et al. proposed CaT: Balanced Continual Graph Learning with Graph Condensation. This approach uses dataset distillation to condense large historical graphs into tiny, highly informative synthetic graph representations. Training exclusively on this balanced memory bank preserves the essential structural properties of past tasks at a fraction of the computational cost.
- Structure Learning: Rather than treating the memory buffer as static, Seungyoon Choi et al. developed DSLR: Diversity Enhancement and Structure Learning for Rehearsal-based Graph Continual Learning. DSLR pairs a coverage-based diversity algorithm to select representative nodes with an explicit Graph Structure Learning (GSL) module. The GSL module actively reconstructs missing, informative topological connections among the buffered nodes, ensuring that message passing during replay mimics the original network dynamics.
- Lossless Synthetic Replay: Similarly, Chaoxi Niu et al. introduced Graph Continual Learning with Debiased Lossless Memory Replay, which replaces traditional node sampling with optimized synthetic node representations and a debiased loss function, preventing the model from becoming overwhelmed by the distribution shifts of incoming data streams.
Parameter Isolation and Structural Freezing
An alternative strategy bypasses replay entirely by physically shielding the neural weights responsible for historical knowledge. If the parameters that encode past topologies are never modified, they cannot be forgotten.
In Continual Learning on Dynamic Graphs via Parameter Isolation, Peiyan Zhang et al. present a framework that identifies stable subgraphs within an evolving network. The model freezes the specific parameters associated with these stable components while dynamically expanding its architecture to allocate new, plastic parameters for incoming streaming data.
Taking this concept to its logical extreme, Giovanni Donghi et al. demonstrate in The Unreasonable Effectiveness of Randomized Representations in Online Continual Graph Learning that freezing the entire structural encoder can be remarkably effective. By utilizing a completely static, randomized backbone to map graph topologies and restricting all online updates strictly to the final classification layer, the system completely isolates structural and backbone drift, offering a highly stable optimization landscape.
For massive, multi-domain scenarios, the authors of AnyGraph: Graph Foundation Model in the Wild scale this isolation principle using a Graph Mixture-of-Experts (MoE) architecture. A lightweight routing mechanism partitions knowledge across distinct topological experts, ensuring that learning the structural patterns of a new domain does not interfere with or overwrite the features learned in another.
Geometric and Semantic Alignment
Instead of physically partitioning parameters, some approaches allow the entire network to remain plastic but enforce geometric or semantic constraints to keep representations stable.
Streaming Data ---> [ Disentanglement Layer ] ---> Component A (Updates with new data)
---> Component B (Preserved historical knowledge)
One way to achieve this is through latent space factorization. In Disentangle-based Continual Graph Representation Learning, Xiaoyu Kou et al. introduce a framework that disentangles node embeddings into multiple independent semantic components. When new streaming data arrives, the model selectively updates only the components directly relevant to the incoming facts, leaving the remaining sub-spaces untouched and preserving historical relationships.
Alternatively, global regularization can prevent the representation space from warping as the graph grows. Yuting Zhang et al. developed SAOT: Self-Supervised Continual Graph Learning with Structure-Aware Optimal Transport, which uses Optimal Transport theory to align the global topology of an evolving graph with its latent representation space. By minimizing the transport distance between the structural graph and the embedding distribution, SAOT prevents structural drift and ensures that the geometric relationships between historical nodes remain consistent over time.
Key Foundational Work
Efficiently embedding dynamic knowledge graphs (2019) — contributed 1 concept elements to this direction.