TL;DR
This paper dives deep into the hardware performance of Dynamic Graph Neural Networks (DGNNs) on CPUs and GPUs, identifying key bottlenecks like temporal data dependencies and data movement. The authors provide a comprehensive analysis and propose optimizations to bridge the gap between algorithmic success and efficient hardware deployment.
What Problem Does This Paper Address?
Dynamic Graph Neural Networks (DGNNs) are powerful tools for analyzing data that changes over time, like social networks or traffic patterns. While algorithms for DGNNs have advanced significantly, deploying them on real-world hardware (CPUs and GPUs) presents a major challenge. Unlike static graphs, DGNNs must handle evolving structures and temporal information, making existing hardware optimizations for static graphs insufficient. This paper aims to bridge this gap by systematically analyzing the performance bottlenecks of popular DGNN models on both CPUs and GPUs, offering insights for future hardware-aware DGNN development.
Methods
The researchers selected eight prominent DGNN models with diverse characteristics and profiled their inference performance on both CPU and GPU. They utilized PyTorch Profiler to analyze memory usage and execution times of different model components and NVIDIA Nsight Systems for a detailed breakdown of CUDA and kernel operations. This profiling allowed them to identify and categorize common hardware bottlenecks. Based on these findings, they propose a set of optimizations spanning software, hardware, and co-design approaches.
Datasets
The paper mentions several datasets used for profiling, including:
- Bitcoin Alpha
- Reddit Hyperlink Network
- Stochastic Block Model
- Wikipedia
- Caltrans Performance Measurement System (PeMS)
- Social Evolution
- GitHub-data
- LastFM
- Molecular Trajectories ISO17 Dataset
Key Insights and Ideas
- Temporal Data Dependency is a Major Hurdle: The inherent sequential nature of processing time-series data in DGNNs, where each step depends on the previous one, severely limits parallelization on hardware, leading to underutilized GPU resources.
- Data Movement is Costly: Frequent transfers of graph topology and node embeddings between CPU and GPU are a significant bottleneck, especially for dynamic graphs where this information changes constantly.
- Workload Imbalance Between CPU and GPU: Graph sampling, often performed on the CPU, can become a bottleneck, leaving the GPU idle while waiting for data. Conversely, large batch sizes can saturate the GPU, leaving the CPU idle.
- GPU Warm-up Overhead is Non-Trivial: Initializing the GPU, loading model weights, and setting up CUDA contexts introduce a significant overhead, particularly for smaller inference tasks.
- Existing Static GNN Optimizations Don't Directly Translate: The unique temporal dynamics of DGNNs require new approaches beyond those developed for static graph processing.
Key Results
The paper provides a detailed breakdown of performance bottlenecks for eight DGNN models. While specific quantitative speedup numbers are not the primary focus, the analysis highlights significant GPU underutilization across various models, often below 1% for some components. For instance, TGAT showed low GPU utilization (5-6%) due to its sequential processing of nodes within a mini-batch. JODIE also exhibited very low GPU utilization (1.5-2.5%) despite using t-batching. The paper illustrates these bottlenecks with detailed profiling results and execution time breakdowns, showing how different modules contribute to the overall inference time.
Limitations and Future Work
The authors acknowledge that this work focuses solely on inference bottlenecks and does not cover the more complex training phase of DGNNs. They suggest that future research should investigate training bottlenecks. Additionally, they propose several optimization strategies but note that these have not yet been experimentally verified. The paper's future work includes profiling DGNN training and designing a dedicated DGNN hardware accelerator to implement the proposed optimizations and push the boundaries of efficient DGNN hardware design.
Summary updated