MEGA: Second-Order Gradient Alignment for Catastrophic Forgetting Mitigation in GFSCIL

Jinhui Pang, Changqing Lin, Hao Lin, Jinglin He, Zhengjun Li, Zhihui Zhang, Xiaoshuai HaoarXiv 2025 · arXiv · OpenReview

Abstract

Graph Few-Shot Class-Incremental Learning (GFSCIL) enables models to continually learn from limited samples of novel tasks after initial training on a large base dataset. Existing GFSCIL approaches typically utilize Prototypical Networks (PNs) for metric-based class representations and fine-tune the model during the incremental learning stage. However, these PN-based methods oversimplify learning via novel query set fine-tuning and fail to integrate Graph Continual Learning (GCL) techniques due to architectural constraints. To address these challenges, we propose a more rigorous and practical setting for GFSCIL that excludes query sets during the incremental training phase. Building on this foundation, we introduce Model-Agnostic Meta Graph Continual Learning (MEGA), aimed at effectively alleviating catastrophic forgetting for GFSCIL. Specifically, by calculating the incremental second-order gradient during the meta-training stage, we endow the model to learn high-quality priors that enhance incremental learning by aligning its behaviors across both the meta-training and incremental learning stages. Extensive experiments on four mainstream graph datasets demonstrate that MEGA achieves state-of-the-art results and enhances the effectiveness of various GCL methods in GFSCIL.

Introduction

In the rapidly evolving world of Graph Neural Networks (GNNs), models often struggle with a "Goldilocks" problem: they are either proficient at learning new information but forget previous knowledge, or they are too rigid to adapt to new data. This challenge is particularly acute in Graph Few-Shot Class-Incremental Learning (GFSCIL), where a model must learn new categories of nodes from just a handful of examples without losing its grip on the "base" knowledge it already possesses.

Existing GFSCIL research suffers from two primary flaws: Query Set Leakage and Architectural Rigidity. Many methods use the query set—data intended for testing—during the incremental training phase to fine-tune the model, creating an unrealistically easy benchmark. Furthermore, reliance on specific structures like Prototypical Networks (PNs) makes it difficult to integrate established Graph Continual Learning (GCL) techniques designed to prevent catastrophic forgetting.

Addressing the "Leakage" Problem

To create a more realistic evaluation environment, the authors propose a rigorous setting that strictly reserves query sets for testing. This prevents models from "cheating" by seeing test data during the training phase, a common issue in prior benchmarks like HAG-Meta and GEOMETER.

Figure 2: Comparison of the proposed rigorous setting in MEGA versus existing GFSCIL methods, highlighting the elimination of query set leakage and increased task complexity.

The MEGA Framework: Meta-Learning for Stability

MEGA (Model-Agnostic Meta Graph Continual Learning) is designed to be architecture-independent, allowing it to be integrated into various GNN backbones. The framework focuses on finding "high-quality priors"—initial model parameters that are naturally robust to a sequence of new tasks.

1. Meta Continual Training Framework (MCTF)

MEGA employs a "double-loop" meta-learning approach. The Inner Loop simulates learning a new task, while the Outer Loop calculates a second-order gradient. This calculation does not merely optimize for the current task; it aligns model behaviors across a sequence of tasks. By optimizing for this alignment during meta-training, the model develops a starting point (prior) that is inherently resistant to forgetting when it encounters real incremental tasks.

Figure 3: The MEGA architecture, illustrating the double-loop Meta-training Stage (Inner and Outer Loops) and the subsequent Incremental Learning Stage.

2. Knowledge Distillation with Single-Instance Replay (KDSIR)

To further bolster stability, MEGA incorporates a lightweight GCL module. Knowledge Distillation (KD) penalizes the current model if its outputs drift too far from the previous version. To manage memory efficiently, Single-Instance Replay (SIR) retains just one representative node from each previous class to "rehearse" during new training sessions.

Experimental Results

MEGA was evaluated on four diverse, large-scale graph datasets: Amazon-Clothing, DBLP, Cora-Full, and Reddit. Key findings include:

  • Accuracy Boost: On the first novel task, MEGA improved accuracy by 2.60% to 7.43% compared to state-of-the-art baselines.
  • Mitigating Forgetting: On the Cora-Full dataset, MEGA maintained a significantly higher "accuracy floor" as more tasks were added, whereas competing models saw performance plummets.
  • Scalability: On the massive Reddit dataset (over 200k nodes), MEGA achieved a base task accuracy of 90.09%, far outperforming competitors like HAG-Meta (76.15%).
  • Linear Complexity: Despite using second-order gradients, MEGA utilizes a gradient-tracking approximation that keeps computational costs linear, ensuring practicality for large-scale graphs.

Conclusion

MEGA introduces a more rigorous and practical paradigm for GFSCIL. By aligning second-order gradients to find stable priors, it effectively mitigates catastrophic forgetting without requiring access to test-time query sets during training. As a model-agnostic framework, MEGA paves the way for more robust and adaptable graph learning systems in real-world, incremental scenarios.

Summary updated

Abstract

Graph Few-Shot Class-Incremental Learning (GFSCIL) enables models to continually learn from limited samples of novel tasks after initial training on a large base dataset. Existing GFSCIL approaches typically utilize Prototypical Networks (PNs) for metric-based class representations and fine-tune the model during the incremental learning stage. However, these PN-based methods oversimplify learning via novel query set fine-tuning and fail to integrate Graph Continual Learning (GCL) techniques due to architectural constraints. To address these challenges, we propose a more rigorous and practical setting for GFSCIL that excludes query sets during the incremental training phase. Building on this foundation, we introduce Model-Agnostic Meta Graph Continual Learning (MEGA), aimed at effectively alleviating catastrophic forgetting for GFSCIL. Specifically, by calculating the incremental second-order gradient during the meta-training stage, we endow the model to learn high-quality priors that enhance incremental learning by aligning its behaviors across both the meta-training and incremental learning stages. Extensive experiments on four mainstream graph datasets demonstrate that MEGA achieves state-of-the-art results and enhances the effectiveness of various GCL methods in GFSCIL. We believe that our proposed MEGA serves as a model-agnostic GFSCIL paradigm, paving the way for future research.
BibTeX
@misc{pang2025mega,
  author = {Jinhui Pang and Changqing Lin and Hao Lin and Jinglin He and Zhengjun Li and Zhihui Zhang and Xiaoshuai Hao},
  title = {{MEGA: Second-Order Gradient Alignment for Catastrophic Forgetting Mitigation in GFSCIL}},
  year = {2025},
  eprint = {2504.13691},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.13691},
}
Figure