Degree-Quant: Quantization-Aware Training for Graph Neural Networks

Shyam A. Tailor, Javier Fernández-Marqués, Nicholas D. Lane

TL;DR

This paper introduces Degree-Quant (DQ), a novel quantization-aware training method for Graph Neural Networks (GNNs) that significantly improves inference efficiency without sacrificing accuracy. DQ addresses unique challenges in quantizing GNNs, enabling up to 4.7x speedups on CPUs with INT8 precision and achieving competitive or superior performance compared to full-precision models, even at INT4 precision.

What Problem Does This Paper Address?

Graph Neural Networks (GNNs) are powerful for modeling complex, irregularly structured data, leading to their widespread use in areas like molecular interactions and social networks. However, making GNNs efficient for inference, especially on resource-constrained devices like smartphones, remains a significant challenge. While quantization (using lower-precision numbers) is a common technique for accelerating neural networks, existing methods often struggle with GNNs, leading to substantial accuracy degradation. This paper aims to bridge this gap by developing a quantization strategy specifically tailored for GNNs that maintains high accuracy while enabling significant performance gains.

Methods

The core contribution is Degree-Quant (DQ), an architecture-agnostic quantization-aware training (QAT) method. DQ tackles two main issues:

  • Stochastic Protection: GNNs are sensitive to quantization errors, particularly at nodes with high in-degrees, due to the aggregation of messages from many neighbors. DQ introduces a stochastic masking mechanism where high-in-degree nodes are more frequently kept at full precision during training. This protects critical parts of the computation from quantization noise, leading to more stable weight updates.
  • Percentile Tracking of Quantization Ranges: Instead of traditional min-max or momentum-based range tracking, DQ uses percentile-based clipping. This method is more robust to the fluctuating aggregation outputs in GNNs, providing more representative quantization ranges and reducing rounding errors, especially at lower bit-widths.

DQ is designed to be compatible with various GNN architectures and can be applied to both weights and activations.

Datasets

The authors evaluated Degree-Quant on six datasets:

  • Citation Networks: Cora, Citeseer (used for node classification)
  • Image Superpixels: MNIST, CIFAR-10 (used for graph classification)
  • Molecular Graphs: ZINC (used for graph regression)
  • Social Networks: REDDIT-BINARY (used for graph classification)

Key Insights and Ideas

  • GNNs have unique quantization challenges: Unlike CNNs, the varying neighborhood sizes and aggregation processes in GNNs introduce specific sources of error that standard quantization techniques struggle to handle. High in-degree nodes are particularly problematic.
  • Topology matters for quantization: The in-degree of nodes significantly impacts the variance of intermediate computations, making some nodes more susceptible to quantization errors. This insight is central to DQ's design.
  • Percentiles are crucial for stable GNN quantization: Traditional range tracking methods are insufficient for GNNs. Percentile-based range estimation provides a more stable and effective way to manage quantization ranges, especially for lower bit-widths.
  • Stochastic protection is key: Protecting high-in-degree nodes during training, even stochastically, dramatically improves the robustness of quantized GNNs.
  • Architecture-agnosticism is achievable: DQ demonstrates that a single, well-designed quantization strategy can be effective across diverse GNN architectures and tasks.

Key Results

  • INT8 Quantization: DQ-trained INT8 GNNs achieve performance comparable to FP32 models across most datasets and architectures.
  • INT4 Quantization: At INT4 precision, DQ models consistently outperform standard QAT baselines, achieving up to 26% gains in accuracy.
  • Speedup: INT8 models with DQ achieve up to 4.7x speedups on CPUs compared to FP32 models.
  • Memory Reduction: INT8 quantization leads to 4-8x reductions in runtime memory usage.
  • Generalization: Unlike previous work, DQ-trained models demonstrate good generalization to unseen graphs.

Limitations and Future Work

  • Training Time: While DQ reduces hyperparameter tuning effort, the percentile operation can add to training time. The authors suggest sampling-based approaches for percentiles to mitigate this.
  • Batch Normalization Folding: The paper notes that folding Batch Normalization layers (a common optimization for quantized models) into weights is left as future work for GNNs.
  • Mixed Precision: While DQ quantizes most components, the paper suggests that not quantizing the final layer (or first layer) can further boost accuracy, especially at INT4.
  • Learned Step Size: The authors propose exploring learned step-size approaches (similar to those used in CNNs) as a potential avenue for further improving GNN quantization performance.
  • Robust Quantization: Investigating methods to make GNN architectures inherently more robust to quantization is also suggested.

Summary updated

Abstract

Graph neural networks (GNNs) have demonstrated strong performance on a wide variety of tasks due to their ability to model non-uniform structured data. Despite their promise, there exists little research exploring methods to make them more efficient at inference time. In this work, we explore the viability of training quantized GNNs, enabling the usage of low precision integer arithmetic during inference. We identify the sources of error that uniquely arise when attempting to quantize GNNs, and propose an architecturally-agnostic method, Degree-Quant, to improve performance over existing quantization-aware training baselines commonly used on other architectures, such as CNNs. We validate our method on six datasets and show, unlike previous attempts, that models generalize to unseen graphs. Models trained with Degree-Quant for INT8 quantization perform as well as FP32 models in most cases; for INT4 models, we obtain up to 26% gains over the baselines. Our work enables up to 4.7x speedups on CPU when using INT8 arithmetic.
BibTeX
@inproceedings{tailor2021degree,
  author = {Shyam A. Tailor and Javier Fernández-Marqués and Nicholas D. Lane},
  title = {{Degree-Quant: Quantization-Aware Training for Graph Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=qKuqJm0FEK},
}
Figure