Many graph models now pair two components in every layer. The first is a local part that passes messages only along real edges, such as chemical bonds. The second is a global part that lets every node attend to every other node. Models built this way, the best known being GraphGPS, are strong when labels are plentiful. Science applications rarely offer that. A chemist may have a few hundred measured molecules, and a peptide assay a few thousand. The usual cheap way to adapt a large pretrained model to such small datasets is prompt tuning: freeze the model and train a few extra input vectors. On graphs, though, prompt tuning has been developed for one half of the hybrid at a time. One line adds learnable tokens to the attention layers of graph Transformers. Another adds learnable vectors to the edges of message-passing networks. Nobody has asked what happens when the frozen model has both halves running side by side.
We propose that, in a parallel hybrid, a prompt placed in one branch only weakly steers the other. Effective few-label adaptation therefore needs prompts in both branches, plus a small learned per-layer gate that decides how much each branch's adapted output counts. If the claim holds, two things should follow. Tasks governed by nearby atoms, such as solubility, should benefit mainly from edge prompts in the local branch. Tasks governed by distant interactions, such as peptide function, should benefit mainly from attention prompts. The gates learned on each task should show which kind of task it is. We test this on a GraphGPS-style model pretrained on PCQM4Mv2 and adapted with 1–10% of labels on MoleculeNet and the Long-Range Graph Benchmark.
Core Claim
Recipe for a General, Powerful, Scalable Graph Transformer (GraphGPS) runs a message-passing network and a global attention module in parallel within each block and sums their outputs. It was studied only under full supervision. Two prompt-tuning results bear directly on what should happen when such a model is frozen.
The first is Deep Prompt Tuning for Graph Transformers (DeepGPT). It prepends trainable "prefix" tokens to the keys and values of every attention layer of a frozen Graphormer pretrained on PCQM4Mv2. This lets the model's attention read extra learned context. With under 0.5% of the parameters, DeepGPT reaches 0.883 AUROC on BACE, against 0.704 for full fine-tuning.
The second is Edge Prompt Tuning for Graph Neural Networks (EdgePrompt). It shows that prompts added to node features get averaged away by message passing. Its fix is to inject prompts into the messages sent along edges, and it reports 62.88% 5-shot accuracy on Cora with a purely local network.
In a parallel hybrid, a prefix token touches only the attention branch. The message-passing branch within the same block never sees it. That branch is affected only indirectly, through the changed node states in later layers, which is exactly the node-level path EdgePrompt shows gets washed out. Edge prompts have the mirror-image limitation: they cannot add long-range context. Our claim is that, at a fixed small parameter budget, combining edge prompts in the local branch, prefix tokens in the global branch, and per-layer gates beats either prompt type alone. We also claim the advantage is largest where a task's dependence on local versus long-range structure differs most from what the model learned in pretraining.
This differs from the existing hybrid-plus-label-efficiency papers in where adaptation happens. Self-Supervised Graph Transformer on Large-Scale Molecular Data (GROVER) and Learning the Neighborhood (C-FREE) both pretrain hybrids. GROVER then fine-tunes the whole model; C-FREE uses linear probes or fine-tuning. Neither adapts the local and global parts separately. A Brain Graph Foundation Model (BrainGFM) adds static graph-prompt vectors trained by meta-learning, but they are not targeted at either branch. HGTDP-DTA uses prompts that change per input, but trains everything end to end on full labels. We make no new pretraining contribution. The pretraining objective is held fixed so that any effect can be attributed to where the adaptation parameters sit.
Research Program
The frozen model. We pretrain a GraphGPS backbone on PCQM4Mv2 HOMO-LUMO gap regression. Its local engine is GINE, a message-passing network that uses edge features. Its global module is either full attention or Performer, a linear-cost approximation to attention, and it uses random-walk structural encodings (features describing each node's local graph shape). We use PCQM4Mv2 because it is DeepGPT's pretraining set, which keeps the comparison with DeepGPT clean. We match Graphormer's scale (about 48M parameters) as closely as compute allows. We will report the gap between this model and DeepGPT's Graphormer rather than hide it.
The adaptation module. Everything below is trained while the backbone stays frozen.
The local branch receives EdgePrompt+-style edge prompts. Each edge's prompt is an attention-weighted mixture of a small bank of shared "anchor" vectors, with weights depending on the edge's two endpoint nodes. The prompt is added inside the GINE message function in every layer.
The global branch receives DeepGPT-style prefix keys and values in every attention layer.
Each layer also gets two learned scalars, $g^{\ell}_{\text{loc}}$ and $g^{\ell}_{\text{glob}}$. They are initialized to 1 and rescale the two branch outputs before the existing sum:
$$h^{\ell+1} = \mathrm{MLP}\big(g^{\ell}_{\text{loc}}\,\mathrm{MPNN}_{\theta}(h^{\ell}; p_E) + g^{\ell}_{\text{glob}}\,\mathrm{Attn}_{\theta}(h^{\ell}; P_{KV})\big)$$
Here $h^{\ell}$ are the node states at layer $\ell$, $\theta$ are the frozen weights, $p_E$ are the edge prompts, and $P_{KV}$ are the prefix keys and values. The task-specific parameters are the edge-prompt anchor bank, the prefix tokens, the two gates per layer, and the prediction head. The total budget is held at or below DeepGPT's roughly 100k parameters. In every comparison, we hold the total budget fixed when parameters are moved between branches.
Analyses. We compute two diagnostics that make the mechanism visible rather than only reporting scores.
The first is a prompt-influence profile. For each prompt type, we measure the norm of the change in each branch's output at each layer when that prompt is switched off. This tests the core assumption directly: prefix tokens should barely move the local branch's output in the same block, and edge prompts should not reach far across the graph.
The second is the learned gate ratio $g_{\text{glob}}/g_{\text{loc}}$, averaged over layers, for each task. We compare it with an independent measure of each task's interaction range. Our main range proxy is the full-supervision gap between GCN and an attention model, which Long Range Graph Benchmark reports on Peptides-func (GCN 0.5930 vs SAN 0.6439 AP). We will measure the same gap ourselves on the other tasks.
Baselines
The naive reference point is the same GraphGPS architecture trained from scratch on each downstream label budget. A second weak reference is a linear probe on the frozen pooled embedding. Together these show what pretraining alone buys before any prompting.
The full fine-tuning baseline updates all backbone weights with a small learning rate, early stopping on validation, and the same tuning budget as every other method. DeepGPT's own result, 0.704 AUROC on BACE against 0.883 for its prefix tuning on Graphormer, predicts that fine-tuning will be weak at small label budgets. Still, it has to be rerun on our backbone rather than assumed.
The strongest existing method is DeepGPT, meaning prefix tokens in the attention layers plus shallow node-input prompts, applied to our frozen GraphGPS model at the same parameter budget. Its reported 0.883 BACE AUROC on Graphormer is the reference number. We will also report DeepGPT on our backbone, because a different backbone could move that number in either direction. Beating it means a statistically significant improvement over DeepGPT-on-GraphGPS, across five scaffold-split seeds, on a majority of tasks at the 1% and 10% label budgets. On long-range tasks, it means matching or exceeding DeepGPT where DeepGPT should be at its strongest. If the combined method beats DeepGPT only on BACE-like local tasks and loses on Peptides, the claim is partially supported, and we will say so.
The second strongest is EdgePrompt+ alone in the local branch, at the same budget. This is the natural extension of the 62.88% Cora 5-shot result to a hybrid model.
Two ablations isolate the key ingredient. The first is gates alone: only the per-layer scalars and the head are trained, with no prompts. It asks whether simply reweighting the frozen branches explains the gains. The second is dual prompts with gates fixed at 1. Comparing it with the full method separates the contribution of the prompts from that of the gates. We also run a capacity-matched control in which the whole budget goes to more prefix tokens, so that "two branches" cannot be confused with "more parameters where they help most."
All methods share the same splits, label subsamples, a hyperparameter search of 30 trials per method per task, and an identical GPU-hour cap.
Why not just…?
Why not just fine-tune the whole pretrained hybrid? DeepGPT's BACE comparison (0.704 vs 0.883) says full fine-tuning overfits small molecular sets. GROVER relied on full fine-tuning and needed heavy regularization on FreeSolv. At 100% labels on the larger tasks, fine-tuning may well win. We will report the label budget at which it overtakes prompting instead of claiming prompting dominates everywhere.
Why not just use DeepGPT's prefix tokens on the hybrid? This is the main threat. Through the residual stream, prefix tokens do eventually reach the local branch in later layers, so they might be enough. The prompt-influence profile measures how much they actually move the local branch's output. The capacity-matched prefix-only control settles the question empirically. If prefix-only matches the dual method on local tasks such as FreeSolv and BACE, our wash-out argument fails for hybrids, and the paper becomes a negative result about edge prompts in hybrid models.
Why not just tune a gate per branch and skip prompts entirely? Reweighting the frozen branches is the cheapest possible way to adapt how much a model relies on local versus long-range structure, and it may capture most of the gain. That is why gates-alone is a full ablation rather than a footnote. If it comes within seed noise of the full method, we will conclude that frozen hybrids already contain the needed features and need only reweighting. That would be a simpler and still useful result.
Why not just pretrain a pure message-passing network and use EdgePrompt? On Peptides-func, GCN trails SAN by about 0.05 AP even with full labels, according to the Long Range Graph Benchmark. Prompts cannot add a receptive field the frozen model lacks. On short-range MoleculeNet tasks, however, a pretrained GINE with EdgePrompt+ may tie the hybrid. We include it as a baseline there, and a tie on those tasks would bound the claim to long-range regimes.
Proposed Experiments
The downstream tasks are BACE, BBBP and ogbg-molhiv (AUROC) and FreeSolv and Lipo (RMSE), all on scaffold splits. We also use Peptides-func (AP) and Peptides-struct (MAE) from the Long Range Graph Benchmark. Each task is run with 1%, 10% and 100% of training labels and five seeds. Full-supervision results provide context only; we found no GraphGPS Peptides number on Lacuna to cite.
The decisive checks are three. First, the dual method beats both single-branch prompt methods at an equal budget on low-label tasks drawn from both the local and the long-range groups. Second, the learned gate ratio orders tasks consistently with their measured GCN-versus-attention gap. Specifically, it is higher on Peptides than on FreeSolv, with a positive rank correlation across all tasks. Third, the prompt-influence profiles show that prefix tokens barely change the local branch's output within the same block, which is the mechanism the whole proposal rests on. If the first check fails while the second holds, the gates alone are the finding. If the third fails, prefix-only tuning is sufficient for hybrids, and that is what we would report.
References
- Recipe for a General, Powerful, Scalable Graph Transformer
- Deep Prompt Tuning for Graph Transformers
- Edge Prompt Tuning for Graph Neural Networks
- Self-Supervised Graph Transformer on Large-Scale Molecular Data
- Learning the Neighborhood: Contrast-Free Multimodal Self-Supervised Molecular Graph Pretraining
- A Brain Graph Foundation Model: Pre-Training and Prompt-Tuning across Broad Atlases and Disorders
- HGTDP-DTA: Hybrid Graph-Transformer with Dynamic Prompt for Drug-Target Binding Affinity Prediction
- Long Range Graph Benchmark