Abstract
Pre-training Graph Neural Networks (GNNs) on large-scale unlabeled data has emerged as a powerful technique for graph-level tasks where labeled data is scarce. However, the subsequent fine-tuning phase—the "last mile" of model adaptation—is typically handled using rigid, "vanilla" strategies. This paper proposes S2PGNN (Search to Fine-tune Pre-trained GNNs), an automated framework that treats fine-tuning as a searchable design space. By optimizing identity augmentation, multi-scale fusion, and graph-level readout, S2PGNN achieves significant performance improvements (9% to 17%) across diverse molecular property prediction benchmarks.
Introduction
In domains like drug discovery and materials science, obtaining labeled molecular data is prohibitively expensive. To address this, researchers leverage self-supervised pre-training, where a GNN learns a "graph grammar" from millions of unlabeled molecules. Despite the success of pre-training, the adaptation of these models to specific downstream tasks often relies on a one-size-fits-all approach: initializing the model with pre-trained weights and training it on a small labeled dataset.
The Problem with Vanilla Fine-tuning
The authors identify three primary reasons why standard fine-tuning is suboptimal:
- Domain Discrepancy: Pre-training data structures may differ significantly from downstream task graphs.
- Overfitting: Small downstream datasets lead to rapid overfitting when using rigid adaptation strategies.
- Structural Rigidity: Different tasks require different information processing (e.g., specific readout or fusion methods), yet standard fine-tuning keeps these components fixed.
Methodology: S2PGNN
S2PGNN transforms the fine-tuning process from a manual choice into an automated search. Instead of a fixed architecture, it explores four critical dimensions of GNN adaptation:
- Identity Augmentation: Uses skip-connections or bottleneck adapters to balance the preservation of pre-trained knowledge with the learning of new task-specific features.
- Multi-scale Fusion: Determines how to aggregate representations from various GNN layers (e.g., concatenation vs. attention-based weighting).
- Graph-level Readout: Selects the optimal pooling method (e.g., sum, max, or set2set) to convert node-level embeddings into graph-level predictions.
- Backbone Convolution: Inherits the message-passing structure from the pre-trained model while allowing weight updates.
Efficient Search Algorithm
To manage the computational complexity of searching thousands of configurations, S2PGNN employs bi-level optimization and differentiable search. By using a continuous relaxation (Gumbel-Softmax), the framework can use gradient descent to find the best configuration. Additionally, candidates share weights within the search space, avoiding the need to train every model from scratch.
Experimental Results
The framework was evaluated on eight molecular property prediction benchmarks from MoleculeNet (including BBBP, Tox21, and ClinTox) using a rigorous scaffold split.
Key Findings
- Consistent Performance Gains: S2PGNN was tested against 10 state-of-the-art pre-training methods (e.g., GraphCL, Mole-BERT). It consistently outperformed vanilla fine-tuning by 9% to 17%.
- Model Agnostic: The search framework proved effective across various backbones, including GIN, GCN, GraphSAGE, and GAT.
- Efficiency: Due to differentiable search and weight sharing, S2PGNN's runtime is comparable to standard regularized fine-tuning baselines.
Conclusion
S2PGNN demonstrates that the strategy used to adapt a pre-trained model is just as critical as the pre-training method itself. By treating fine-tuning as a "first-class citizen" and allowing for data-aware adaptation, S2PGNN provides a robust path for improving GNN performance on specialized tasks. Future work may extend this framework to node-level and link-level tasks or explore the transferability of discovered strategies across similar datasets.
Summary updated