TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs

Yuke Wang, Boyuan Feng, Zheng Wang, Guyue Huang, Yufei Ding

Abstract

Recently, graph neural networks (GNNs), as the backbone of graph-based machine learning, demonstrate great success in various domains. However, the performance of GNNs is usually unsatisfactory due to the highly sparse and irregular graph-based operations. To this end, we propose TC-GNN, the first GNN acceleration framework based on GPU Tensor Core Units (TCUs). The core idea is to reconcile the "Sparse" GNN computation with the high-performance "Dense" TCUs. Specifically, we conduct an in-depth analysis of the sparse operations in mainstream GNN computing frameworks. We introduce a novel sparse graph translation technique to facilitate TCU processing of the sparse GNN workload. We implement an effective CUDA core and TCU collaboration design to fully utilize GPU resources. We integrate TC-GNN with the PyTorch framework for high programmability. Rigorous experiments show an average of 1.70X speedup over the state-of-the-art DGL framework across various models and datasets.

Introduction

Graph Neural Networks (GNNs) have emerged as the gold standard for learning on relational data, from social networks to molecular structures. However, a fundamental hardware mismatch exists: GNNs are inherently sparse and irregular, while modern GPU accelerators—specifically Tensor Core Units (TCUs)—are optimized for dense, rigid matrix mathematics. TC-GNN bridges this gap, enabling GNNs to finally harness the massive throughput of Tensor Cores.

The Challenge of Sparse GNNs

Modern GPUs feature two primary computing resources: flexible CUDA cores for general tasks and specialized Tensor Cores for high-speed dense matrix multiplication. While deep learning has leveraged Tensor Cores for CNNs and Transformers, GNNs have struggled to do so.

The primary bottleneck in GNNs is the Aggregation phase, where a node fetches data from its neighbors. As shown in Figure 1, GNN computation typically involves alternating between neighbor aggregation and node updates. Because graph connections are irregular, these operations result in sparse matrix computations. Running these directly on Tensor Cores leads to significant inefficiencies, such as memory waste from zero-padding or high overhead from indirect memory access.

GNN computation phases: Neighbor Aggregation and Node Update.

Methodology: Reconciling Sparsity and Density

TC-GNN introduces two primary innovations to adapt sparse graph workloads for dense hardware.

1. Sparse Graph Translation (SGT)

The core of TC-GNN is Sparse Graph Translation. Most GNN operations can be categorized as Sparse Matrix-Matrix Multiplication (SpMM) or Sampled Dense-Dense Matrix Multiplication (SDDMM), as illustrated in Figure 2. SGT identifies nodes with similar neighbor sets and remaps neighbor IDs into dense blocks (TC_blocks). These blocks are designed to fit the rigid input requirements of Tensor Cores (e.g., 16x16 tiles). This densification reduces the total number of tiles processed by an average of 67%.

Mapping graph operations to sparse matrix computations (SpMM and SDDMM).

2. CUDA Core and TCU Collaboration

The framework employs a two-level workload decomposition:

  • CUDA Cores manage the irregular tasks: loading data from global memory, organizing it in shared memory, and handling indices.
  • Tensor Cores perform the heavy-duty matrix multiplications on the densified tiles prepared by the CUDA cores.

This collaboration allows TC-GNN to reach approximately 85% SM occupancy, significantly improving cache hit rates.

Evaluation and Results

TC-GNN was evaluated across diverse datasets, including small/dense benchmarks (Cora, PPI), graph kernels (PROTEINS, DD), and large-scale irregular graphs (Amazon0505, BlogCatalog).

  • Performance: TC-GNN achieves an average 1.70× speedup over the Deep Graph Library (DGL) and significantly outperforms PyTorch Geometric (PyG).
  • Efficiency: Compared to specialized baselines like cuSPARSE bSpMM, TC-GNN is 1.76× faster on neighbor aggregation.
  • Scalability: The framework scales effectively with increasing node embedding dimensions (from 16 to 256), proving its utility for modern high-dimensional AI models.
  • Overhead: The SGT preprocessing step is a one-time cost, typically accounting for only ~4% of total training time, which is easily offset by faster training epochs.

Conclusion

TC-GNN demonstrates that the "Sparse" nature of GNNs can be reconciled with "Dense" hardware through intelligent data translation and resource collaboration. By providing a PyTorch-integrated backend, TC-GNN makes high-performance GNN training accessible to the broader research community without requiring low-level hardware expertise.

Summary updated

Abstract

Recently, graph neural networks (GNNs), as the backbone of graph-based machine learning, demonstrate great success in various domains (e.g., e-commerce). However, the performance of GNNs is usually unsatisfactory due to the highly sparse and irregular graph-based operations. To this end, we propose TC-GNN, the first GNN acceleration framework based on GPU Tensor Core Units (TCUs). The core idea is to reconcile the "Sparse" GNN computation with the high-performance "Dense" TCUs. Specifically, we conduct an in-depth analysis of the sparse operations in mainstream GNN computing frameworks. We introduce a novel sparse graph translation technique to facilitate TCU processing of the sparse GNN workload. We implement an effective CUDA core and TCU collaboration design to fully utilize GPU resources. We integrate TC-GNN with the PyTorch framework for high programmability. Rigorous experiments show an average of 1.70X speedup over the state-of-the-art DGL framework across various models and datasets.
BibTeX
@inproceedings{wang2023tc,
  author = {Yuke Wang and Boyuan Feng and Zheng Wang and Guyue Huang and Yufei Ding},
  title = {{TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs}},
  booktitle = {USENIX ATC},
  year = {2023},
  url = {https://openreview.net/forum?id=ibrT192ArX},
}
Figure