Continuous Spiking Graph Neural Networks

Nan Yin, Mengzhu Wang, Li Shen, Hitesh Laxmichand Patel, Baopu Li, Bin Gu, Huan Xiong

Abstract

Continuous graph neural networks (CGNNs) have garnered significant attention due to their ability to generalize existing discrete graph neural networks (GNNs) by introducing continuous dynamics. They typically draw inspiration from diffusion-based methods to introduce a novel propagation scheme, which is analyzed using ordinary differential equations (ODE). However, the implementation of CGNNs requires significant computational power, making them challenging to deploy on battery-powered devices. Inspired by recent spiking neural networks (SNNs), which emulate a biological inference process and provide an energy-efficient neural architecture, we incorporate the SNNs with CGNNs in a unified framework, named Continuous Spiking Graph Neural Networks (COS-GNN). We employ SNNs for graph node representation at each time step, which are further integrated into the ODE process along with time. To enhance information preservation and mitigate information loss in SNNs, we introduce the high-order structure of COS-GNN, which utilizes the second-order ODE for spiking representation and continuous propagation. Moreover, we provide the theoretical proof that COS-GNN effectively mitigates the issues of exploding and vanishing gradients, enabling us to capture long-range dependencies between nodes. Experimental results on graph-based learning tasks demonstrate the effectiveness of the proposed COS-GNN over competitive baselines.

Introduction

Graph Neural Networks (GNNs) have become the standard for modeling relational data. However, modern architectures face a critical trade-off between modeling power and computational efficiency. Continuous GNNs (CGNNs) utilize Ordinary Differential Equations (ODEs) to model smooth changes in node representations, offering superior performance but at a high energy cost that limits deployment on edge devices. Conversely, Spiking Neural Networks (SNNs) offer a biologically inspired, energy-efficient alternative by using discrete spikes. However, the binarization process in SNNs often leads to significant information loss, compromising accuracy.

To address these challenges, we propose Continuous Spiking Graph Neural Networks (COS-GNN). This framework unifies the energy efficiency of SNNs with the continuous modeling capabilities of CGNNs by treating the system as a set of Partial Differential Equations (PDEs).

Methodology: Bridging Spikes and Continuous Dynamics

The core of COS-GNN lies in its integration of two distinct temporal dimensions: the latency dimension ($N$) of the SNN and the evolution time dimension ($T$) of the continuous graph propagation.

Dual-Time Integration

The model operates by employing SNNs for node representation at each discrete time step, which are then integrated into a continuous ODE process. This "dual-time" approach allows the model to capture both the internal spiking dynamics of individual nodes and the global evolution of the graph structure over time.

Overview of the COS-GNN framework integrating spiking processes with continuous graph dynamics.

Second-Order Structure

To mitigate the information loss inherent in standard SNN binarization, we introduce a high-order structure (COS-GNN-2nd). Inspired by the gradual release of neurotransmitters in biological brains, this version utilizes second-order ODEs—analogous to damped oscillators—for both node representation and propagation. This higher-order dynamics acts as a buffer, preserving fine-grained correlations that are often lost in first-order spiking models.

Theoretical Stability

A key theoretical contribution of this work is the proof that the second-order structure in COS-GNN effectively mitigates exploding and vanishing gradients. By stabilizing the gradient flow, the model can capture long-range dependencies between nodes, a common failure point for traditional message-passing GNNs.

Experimental Evaluation

We evaluated COS-GNN across a variety of graph learning tasks, including homophilic node classification (Cora, Citeseer, Pubmed), heterophilic node classification (Texas, Wisconsin, Cornell), and graph classification (MNIST-75).

Performance and Efficiency

As shown in the experimental results, COS-GNN-2nd consistently outperforms both discrete GNNs and existing continuous models. On the Texas dataset, for instance, COS-GNN-2nd achieved 87.3% accuracy, surpassing its closest continuous competitor by over 5%.

Performance comparison of COS-GNN against baseline models on homophilic node classification datasets.

Beyond accuracy, the efficiency gains are substantial. On the Cora dataset, a standard Graph Attention Network (GAT) requires approximately 308.94K operations. In contrast, COS-GNN-1st requires only 2.33K operations—a reduction of over 130x. This efficiency stems from the sparse, event-driven nature of spiking representations.

Conclusion

COS-GNN represents a significant step toward energy-efficient, high-performance graph modeling. By framing the interaction between spiking latency and graph evolution as a PDE and introducing a second-order structure, we provide a mathematically robust framework that preserves information while minimizing computational overhead. Future work will explore the deployment of COS-GNN on specialized neuromorphic hardware to fully realize its energy-saving potential and its application to large-scale dynamic social and traffic networks.

Summary updated

Abstract

Continuous graph neural networks (CGNNs) have garnered significant attention due to their ability to generalize existing discrete graph neural networks (GNNs) by introducing continuous dynamics. They typically draw inspiration from diffusion-based methods to introduce a novel propagation scheme, which is analyzed using ordinary differential equations (ODE). However, the implementation of CGNNs requires significant computational power, making them challenging to deploy on battery-powered devices. Inspired by recent spiking neural networks (SNNs), which emulate a biological inference process and provide an energy-efficient neural architecture, we incorporate the SNNs with CGNNs in a unified framework, named Continuous Spiking Graph Neural Networks (COS-GNN). We employ SNNs for graph node representation at each time step, which are further integrated into the ODE process along with time. To enhance information preservation and mitigate information loss in SNNs, we introduce the high-order structure of COS-GNN, which utilizes the second-order ODE for spiking representation and continuous propagation. Moreover, we provide the theoretical proof that COS-GNN effectively mitigates the issues of exploding and vanishing gradients, enabling us to capture long-range dependencies between nodes. Experimental results on graph-based learning tasks demonstrate the effectiveness of the proposed COS-GNN over competitive baselines.
BibTeX
@misc{yin2024continuous,
  author = {Nan Yin and Mengzhu Wang and Li Shen and Hitesh Laxmichand Patel and Baopu Li and Bin Gu and Huan Xiong},
  title = {{Continuous Spiking Graph Neural Networks}},
  year = {2024},
  eprint = {2404.01897},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.01897},
}
Figure