G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks

Anchun Gui, Jinqiang Ye, Han XiaoAAAI 2024 · DOI

G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks

Graph Transformer Networks (GTNs) have shown immense potential in graph-based tasks, yet their large parameter counts make full fine-tuning computationally prohibitive. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA and Adapters work well in NLP, they often underperform on graphs due to a lack of structural awareness and significant feature distribution shifts. G-Adapter is a novel PEFT framework that incorporates graph structural information directly into adapter modules and utilizes specialized optimization to stabilize training.

The Problem: Feature Distribution Shift

The "pre-train then fine-tune" paradigm is standard in modern AI, but fine-tuning massive GTNs for every downstream task is expensive. Existing PEFT methods like LoRA or standard Adapters struggle with graphs because they treat graph nodes as independent tokens, ignoring the structural connectivity (the adjacency matrix). Consequently, the learned features drift away from the optimal representations established during pre-training. This leads to a significant performance gap compared to full fine-tuning, especially on complex molecular datasets.

Methodology: Making PEFT "Graph-Aware"

G-Adapter addresses these issues through two primary innovations:

  1. Structure-Aware Bottleneck Design: Standard adapters use simple linear layers. G-Adapter replaces these with a Graph Convolution (GCN) operation that utilizes the graph's adjacency matrix or shortest-path distances as an inductive bias. This ensures the adapter considers local neighborhood information during updates. To maintain efficiency, a low-rank decomposition ($W_{down}$ and $W_{up}$) is employed to minimize the number of new parameters.
  2. Bregman Proximal Point Optimization: To mitigate aggressive updates and distribution shift, a specialized optimizer is used. It incorporates a regularization term based on symmetric KL-divergence, ensuring the model's output distributions remain close to the pre-trained starting point while optimizing for the new task.
Figure 6: G-Adapter maintains a feature distribution close to full fine-tuning, minimizing the distribution shift compared to other PEFT methods.

Experimental Results

G-Adapter was evaluated on nine benchmarks, including large-scale molecular datasets like MolHIV and MolPCBA, using Graphormer and MAT backbones.

Figure 3: Performance gap relative to full fine-tuning across different ratios of trainable parameters. G-Adapter (red stars) consistently outperforms other PEFT methods and matches or exceeds full fine-tuning performance with minimal parameters.

Key Findings: * High Efficiency: On the MolHIV dataset, G-Adapter achieved a 0.790 AUC using only 0.24% of the trainable parameters, compared to full fine-tuning's 0.804 AUC at 100% parameters. * Superiority on Small Data: On the Estrogen-β dataset, G-Adapter outperformed full fine-tuning by 1.3% AUC. Freezing the majority of the model acts as a powerful regularizer, preventing the "catastrophic forgetting" often seen when training large models on small datasets. * Resource Savings: G-Adapter requires only 0.4MB of storage per task checkpoint (vs. 161MB for full fine-tuning) and reduces GPU memory consumption by over 50% during training.

Conclusion

G-Adapter demonstrates that re-introducing structural inductive biases is critical for effective PEFT in graph domains. By combining structure-aware layers with distribution-preserving optimization, it bridges the gap between efficiency and performance. Future research could explore more sophisticated structural encodings, such as Laplacian eigenvectors or complex motifs, to further enhance performance on massive-scale datasets.

Summary updated

BibTeX
@inproceedings{gui2024g,
  author = {Anchun Gui and Jinqiang Ye and Han Xiao},
  title = {{G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i11.29112},
}
Figure