Abstract
Graph Transformers are powerful tools for modeling complex graph structures, but their adaptation to downstream tasks via full fine-tuning is computationally expensive and prone to overfitting on small datasets. We present Deep Graph Prompt Tuning (DeepGPT), a parameter-efficient alternative that freezes the pre-trained backbone and optimizes only a minimal set of task-specific "prompt" tokens. By injecting these tokens at both the input level and within every transformer layer, DeepGPT matches or exceeds the performance of traditional fine-tuning while utilizing less than 0.5% of the task-specific parameters and achieving faster convergence.
Introduction: The "Large Model Tax" in Graph Learning
While Graph Transformers excel at capturing long-range dependencies in molecular and social graphs, they present significant challenges for practical deployment. Fine-tuning these massive models for every new task requires substantial compute and storage resources. Furthermore, in specialized scientific domains where datasets are often small, full fine-tuning frequently leads to overfitting.
Inspired by the success of prompt tuning in Natural Language Processing, DeepGPT seeks to steer pre-trained graph models using learned vectors rather than full parameter updates. This approach treats the pre-trained model as a fixed "knowledge base" that is navigated by lightweight prompts.
Methodology: Deep vs. Shallow Prompting
DeepGPT distinguishes itself from previous efforts by employing a two-tiered prompting strategy:
- Shallow Graph Prompts: A trainable vector is added to each node's feature vector at the input level, providing a global context for the entire graph.
- Deep Prefix Tokens: Trainable "prefix" tokens are prepended to hidden embeddings at every layer of the transformer. This allows the model to re-contextualize its pre-trained knowledge at multiple levels of abstraction.
By freezing the entire pre-trained backbone, DeepGPT significantly reduces the memory footprint during training. Only the lightweight prompt tokens and the final classification head are updated.
Experimental Setup
The researchers evaluated DeepGPT using models pre-trained on PCQM4Mv2, a large-scale quantum chemistry dataset containing over 3.7 million molecules. Evaluation spanned eleven benchmarks from MoleculeNet and the Open Graph Benchmark (OGB), covering both classification (e.g., BACE, BBBP, MOLHIV) and regression tasks (e.g., FreeSolv, Lipo).
Key Insights and Results
Performance and Parameter Efficiency
DeepGPT demonstrated remarkable performance, often outperforming full fine-tuning (FT) on smaller datasets where FT is prone to overfitting. For instance, on the BACE dataset with the Graphormer architecture, DeepGPT achieved an AUROC of 0.883, a substantial improvement over FT's 0.704. This success is achieved with extreme efficiency; for a 48-million-parameter Graphormer model, DeepGPT requires only 100k task-specific parameters (<0.5%).
Faster Convergence
Beyond parameter efficiency, DeepGPT typically reaches peak performance in fewer training cycles. As shown in the comparison across different architectures, the number of epochs required for convergence is significantly reduced when using prompt tuning.
The Importance of Depth
Ablation studies highlighted that "deep" prompting is critical. Injecting prompts into middle layers often yielded better results than focusing solely on the input or output layers, suggesting that middle layers contain the most flexible semantic information for task adaptation.
Limitations and Future Outlook
Despite its efficiency, DeepGPT faces some implementation challenges. In certain architectures like LiGhT, the overhead of token concatenation can increase the duration of individual training epochs, even if the total number of epochs is lower. Future work will explore optimizing prompt "shapes" and extending this approach to linear transformer variants to better handle massive-scale networks.
Summary updated