Degree-Quant: Quantization-Aware Training for Graph Neural Networks

Shyam Anil Tailor, Javier Fernández-Marqués, Nicholas Donald Lane

TL;DR

Degree-Quant (DQ) is a novel quantization-aware training method that enables Graph Neural Networks (GNNs) to operate using low-precision integers (INT8/INT4) instead of standard floating-point numbers. By stochastically protecting high-degree nodes during training and utilizing percentile-based range tracking, DQ achieves up to 4.7× speedups on CPUs and significant memory reductions with almost no loss in accuracy.

The Efficiency Challenge of GNNs

Graph Neural Networks (GNNs) are highly effective for modeling non-uniform structured data, such as social networks, molecular structures, and recommendation systems. However, they face a unique efficiency hurdle: unlike standard Convolutional Neural Networks (CNNs) where computation is fixed by image resolution, a GNN's computational load scales directly with the size and density of the input graph.

Computational operations of GNNs compared to standard CNNs across various datasets.

For instance, while a tiny 2-layer GNN might only take up 81KB of storage, processing a massive graph like Reddit can require tens of billions of operations. To deploy these models on edge devices or smartphones, quantization—converting 32-bit floating-point numbers (FP32) into 8-bit or 4-bit integers—is essential.

Unfortunately, standard quantization methods that work perfectly for CNNs often fail for GNNs. The authors identify the primary culprit: node degree. In any given graph, certain "hub" nodes possess significantly more connections than others. During a GNN's aggregation phase, these high-degree nodes produce output values with drastically higher means and variances. These extreme outliers distort the quantization ranges, effectively ruining the precision for the rest of the network.

The Degree-Quant Method

To solve this, the authors propose Degree-Quant (DQ), an architecture-agnostic approach designed to stabilize GNN quantization during training. DQ relies on two core mechanisms:

Degree-Quant stochastically protects high-degree nodes (blue circles) in full precision while quantizing the rest (red squares).
  1. Stochastic Protection of High-Degree Nodes: During the training phase, the model randomly "protects" specific nodes, computing their features in full precision (FP32). Crucially, the probability of a node being protected is proportional to its degree. This ensures that the gradients flowing backward through the network's most influential hubs remain accurate and are not corrupted by low-precision rounding noise. At inference time, this protection is disabled, and the entire network runs efficiently at low precision.
  2. Percentile Tracking of Quantization Ranges: Traditional quantization tracks the absolute minimum and maximum values to establish the numeric "range." Because GNNs generate extreme outliers at high-degree nodes, this min/max range becomes excessively wide, leaving insufficient resolution for the vast majority of normal data points. DQ instead employs percentile tracking (clipping the top and bottom 0.1% of values) to establish a much more representative range, drastically reducing rounding errors.

Key Insights and Performance

The researchers validated Degree-Quant across six diverse datasets, including citation networks (Cora, CiteSeer), computer vision superpixels (MNIST, CIFAR-10), molecules (ZINC), and social networks (REDDIT-BINARY).

  • Aggregation is the Bottleneck: The paper mathematically and empirically demonstrates that the variance of aggregated messages grows with node degree. Quantization fails in GNNs not because of the model weights, but due to the structural irregularity of the input graphs themselves.
  • Strong Generalization: Unlike some prior quantization attempts, models trained with DQ generalize successfully to unseen graphs, making them viable for real-world deployment.
  • Accuracy and Speed: For INT8 quantization, DQ-trained models perform just as well as—and occasionally better than—their full-precision FP32 counterparts. For ultra-low INT4 quantization, DQ delivers up to 26% accuracy gains over standard quantization baselines. Furthermore, the authors demonstrated a 4.7× speedup on a 22-core Intel CPU using INT8 arithmetic, alongside a 4–8× reduction in runtime memory usage.

Limitations and Future Work

While highly effective, the method does have limitations. Training quantized GNNs can be unstable; some experiments exhibited standard deviations as high as 18%, and certain models struggled to converge on GPUs, requiring CPU training. Additionally, while CPU speedups were substantial, GPU speedups were more modest (~1.3×) because current GPU kernels are not yet fully optimized for the specific sparse operations inherent to GNNs. Finally, adapting "Batch Normalization folding" for graph structures remains an open challenge for future research.

Summary updated

Abstract

Graph neural networks (GNNs) have demonstrated strong performance on a wide variety of tasks due to their ability to model non-uniform structured data. Despite their promise, there exists little research exploring methods to make them more efficient at inference time. In this work, we explore the viability of training quantized GNNs, enabling the usage of low precision integer arithmetic during inference. For GNNs seemingly unimportant choices in quantization implementation cause dramatic changes in performance. We identify the sources of error that uniquely arise when attempting to quantize GNNs, and propose an architecturally-agnostic and stable method, Degree-Quant, to improve performance over existing quantization-aware training baselines commonly used on other architectures, such as CNNs. We validate our method on six datasets and show, unlike previous quantization attempts, that models generalize to unseen graphs. Models trained with Degree-Quant for INT8 quantization perform as well as FP32 models in most cases; for INT4 models, we obtain up to 26% gains over the baselines. Our work enables up to 4.7x speedups on CPU when using INT8 arithmetic.
BibTeX
@inproceedings{tailor2021degree,
  author = {Shyam Anil Tailor and Javier Fernández-Marqués and Nicholas Donald Lane},
  title = {{Degree-Quant: Quantization-Aware Training for Graph Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=qKuqJm0FEK},
}
Figure