Deep Prompt Tuning for Graph Transformers

Reza Shirkavand, Heng HuangarXiv 2023 · DOI · arXiv

Abstract

Graph Transformers are powerful tools for modeling complex graph structures, but their adaptation to downstream tasks via full fine-tuning is computationally expensive and prone to overfitting on small datasets. We present Deep Graph Prompt Tuning (DeepGPT), a parameter-efficient alternative that freezes the pre-trained backbone and optimizes only a minimal set of task-specific "prompt" tokens. By injecting these tokens at both the input level and within every transformer layer, DeepGPT matches or exceeds the performance of traditional fine-tuning while utilizing less than 0.5% of the task-specific parameters and achieving faster convergence.

Introduction: The "Large Model Tax" in Graph Learning

While Graph Transformers excel at capturing long-range dependencies in molecular and social graphs, they present significant challenges for practical deployment. Fine-tuning these massive models for every new task requires substantial compute and storage resources. Furthermore, in specialized scientific domains where datasets are often small, full fine-tuning frequently leads to overfitting.

Inspired by the success of prompt tuning in Natural Language Processing, DeepGPT seeks to steer pre-trained graph models using learned vectors rather than full parameter updates. This approach treats the pre-trained model as a fixed "knowledge base" that is navigated by lightweight prompts.

Methodology: Deep vs. Shallow Prompting

DeepGPT distinguishes itself from previous efforts by employing a two-tiered prompting strategy:

  1. Shallow Graph Prompts: A trainable vector is added to each node's feature vector at the input level, providing a global context for the entire graph.
  2. Deep Prefix Tokens: Trainable "prefix" tokens are prepended to hidden embeddings at every layer of the transformer. This allows the model to re-contextualize its pre-trained knowledge at multiple levels of abstraction.
Architecture comparison between traditional fine-tuning and Deep Graph Prompt Tuning (DeepGPT).

By freezing the entire pre-trained backbone, DeepGPT significantly reduces the memory footprint during training. Only the lightweight prompt tokens and the final classification head are updated.

Experimental Setup

The researchers evaluated DeepGPT using models pre-trained on PCQM4Mv2, a large-scale quantum chemistry dataset containing over 3.7 million molecules. Evaluation spanned eleven benchmarks from MoleculeNet and the Open Graph Benchmark (OGB), covering both classification (e.g., BACE, BBBP, MOLHIV) and regression tasks (e.g., FreeSolv, Lipo).

Key Insights and Results

Performance and Parameter Efficiency

DeepGPT demonstrated remarkable performance, often outperforming full fine-tuning (FT) on smaller datasets where FT is prone to overfitting. For instance, on the BACE dataset with the Graphormer architecture, DeepGPT achieved an AUROC of 0.883, a substantial improvement over FT's 0.704. This success is achieved with extreme efficiency; for a 48-million-parameter Graphormer model, DeepGPT requires only 100k task-specific parameters (<0.5%).

Faster Convergence

Beyond parameter efficiency, DeepGPT typically reaches peak performance in fewer training cycles. As shown in the comparison across different architectures, the number of epochs required for convergence is significantly reduced when using prompt tuning.

Average number of epochs to converge across different Graph Transformer models, comparing Fine-tuning and DeepGPT.

The Importance of Depth

Ablation studies highlighted that "deep" prompting is critical. Injecting prompts into middle layers often yielded better results than focusing solely on the input or output layers, suggesting that middle layers contain the most flexible semantic information for task adaptation.

Limitations and Future Outlook

Despite its efficiency, DeepGPT faces some implementation challenges. In certain architectures like LiGhT, the overhead of token concatenation can increase the duration of individual training epochs, even if the total number of epochs is lower. Future work will explore optimizing prompt "shapes" and extending this approach to linear transformer variants to better handle massive-scale networks.

Summary updated

Abstract

Graph transformers have gained popularity in various graph-based tasks by addressing challenges faced by traditional Graph Neural Networks. However, the quadratic complexity of self-attention operations and the extensive layering in graph transformer architectures present challenges when applying them to graph based prediction tasks. Fine-tuning, a common approach, is resource-intensive and requires storing multiple copies of large models. We propose a novel approach called deep graph prompt tuning as an alternative to fine-tuning for leveraging large graph transformer models in downstream graph based prediction tasks. Our method introduces trainable feature nodes to the graph and pre-pends task-specific tokens to the graph transformer, enhancing the model's expressive power. By freezing the pre-trained parameters and only updating the added tokens, our approach reduces the number of free parameters and eliminates the need for multiple model copies, making it suitable for small datasets and scalable to large graphs. Through extensive experiments on various-sized datasets, we demonstrate that deep graph prompt tuning achieves comparable or even superior performance to fine-tuning, despite utilizing significantly fewer task-specific parameters. Our contributions include the introduction of prompt tuning for graph transformers, its application to both graph transformers and message passing graph neural networks, improved efficiency and resource utilization, and compelling experimental results. This work brings attention to a promising approach to leverage pre-trained models in graph based prediction tasks and offers new opportunities for exploring and advancing graph representation learning.
BibTeX
@misc{shirkavand2023deep,
  author = {Reza Shirkavand and Heng Huang},
  title = {{Deep Prompt Tuning for Graph Transformers}},
  year = {2023},
  eprint = {2309.10131},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.10131},
}
Figure