Threshold Tuning: Few-Shot Adaptation of Graph Encoders by Learning Where Neurons Stay Silent

Most graph neural networks, meaning models that update each node's representation by repeatedly aggregating messages from its neighbors, already use ReLU, a non-linearity that outputs exact zeros for negative inputs. Every zero in a node's hidden vector is a message that never has to be multiplied, stored, or sent to a neighbor. The proposal is to adapt a pre-trained graph encoder to a new task with only one to five labels per class by training almost nothing except a per-channel firing threshold. That threshold decides, for each hidden feature, which nodes are allowed to emit it.

The same small set of parameters then does two jobs. It restricts the model to a small, task-selected subset of pre-trained features, which should resist overfitting when labels are scarce. It also makes node representations sparse, so neighbor aggregation, the memory needed to store hidden states during full-graph inference, and the data exchanged in distributed training all shrink roughly in proportion to the fraction of neurons left firing. The bet is that on graphs, unlike in language models, adaptation, regularization, and efficiency can be bought with the same knob.

Core Claim

A frozen ReLU graph encoder can be adapted to few-shot node classification by learning one threshold per hidden channel per layer, together with a linear head. The thresholds are trained under a sparsity budget that is measured on the entire unlabeled graph rather than on the handful of labeled nodes. We claim this matches or beats input-level prompting of a frozen backbone on standard few-shot node classification. We further claim that at a firing rate of roughly 10–20% it cuts the cost of neighbor aggregation, and the bytes of hidden state that must be kept in memory or communicated, by a comparable factor.

The distinctive element is the budget signal. With five labels, learning even a few hundred thresholds from labels alone would overfit. The requirement that each channel fire for only a fraction $\rho$ of the graph's nodes, however, is estimated from thousands to millions of unlabeled nodes. It pins down the thresholds far more tightly than the labels do.

Concretely, at layer $l$ and channel $c$ we replace $\text{ReLU}(x)$ with the truncating gate $g(x) = x \cdot \mathbb{1}[x > t_{l,c}]$. This gate zeroes small positive values without shifting large ones, following the thresholded ReLU of ProSparse. We train the thresholds $t$ and the head with

$$ \mathcal{L}(t) = \frac{1}{|S|}\sum_{v \in S} \text{CE}(f_t(v), y_v) + \lambda \sum_{l,c} \big[\hat{r}_{l,c}(t) - \rho\big]_+^2 , $$

where $S$ is the labeled support set, $y_v$ is the label of node $v$, $f_t$ is the gated encoder plus head, and CE is cross-entropy. The quantity $\hat{r}_{l,c}(t)$ is the fraction of all graph nodes on which channel $c$ of layer $l$ fires, estimated on a random node sample. $\lambda$ weights the budget term, and gradients pass through the hard gate with a straight-through estimator.

The efficiency claim follows from how aggregation is computed. Aggregating channel $c$ requires work only on edges whose source node is active in $c$. The multiply-accumulates in sparse aggregation therefore scale as $\sum_{l,c} |\{(u,v) \in E : h^{(l)}_u[c] \neq 0\}| \approx \rho\,|E|\,d$ per layer, where $E$ is the edge set and $d$ the hidden width. This is an expected-value statement that holds when firing is not concentrated on high-degree nodes, and we will measure whether it is.

Where This Sits

The mechanism is borrowed from language models. ReLU Strikes Back shows that swapping smooth activations for ReLU in LLMs yields over 90% activation sparsity without losing capacity. ProSparse reaches about 89% sparsity on LLaMA2-7B with progressive activation penalties and a fixed threshold, and gets up to 4.52x inference speedup. ProSparse also constrains us: converting a dense-activation model required continual training on large text corpora. That is impossible with five labels.

Our response is to avoid the conversion. We pretrain the graph encoder with ReLU from the start, so the few-shot stage only moves thresholds on an already-sparse model. Two further results suggest pretraining will hand us sparse representations. The Lazy Neuron Phenomenon reports that trained Transformer MLPs end with under 6% of neurons firing per token. The Sparsing Law finds that sparsity in ReLU networks grows with training data. Neither has been checked for message-passing encoders, which is our first measurement.

Two lines of work motivate the regularization hypothesis without testing it on graphs. Why Regularized Auto-Encoders learn Sparse Representation? explains how penalties push pre-activations below a zero-saturating threshold, which is the mechanism our budget term relies on. The Utility of Sparse Representations for Control in Reinforcement Learning shows that activation-level penalties cut representation overlap between states from 54.0 to 8.8 in Puddle World, while weight decay and dropout did not localize features. We carry that contrast into few-shot graph transfer, where less overlap between classes should mean less memorization of the support set.

The closest adaptation methods for graphs change weights or inputs, not activations. Sparsity-promoting fine-tuning of equivariant materials foundation models learns soft thresholds on weight updates and adapts with 0.5–3% of parameters. Its authors state that the forward pass stays dense, with no memory or speed benefit. We threshold activations instead of weights, and aim for exactly the forward-pass savings they leave on the table.

Virtual Node Tuning is our main competitor. It freezes a pre-trained Graph Transformer and learns input-layer "virtual node" prompts for 1–5-shot classes, reaching 76.68% on 5-way 1-shot CoraFull. It also shows that full fine-tuning on so few nodes degrades badly, which rules out full fine-tuning as our design. Its stated weakness is that prompt influence fades across deep layers. Thresholds act at every layer, so they sidestep that problem, but they cannot add new directions to the representation the way a prompt can. Which effect matters more is an empirical question we will settle.

On scalability, existing methods remove computation statically or ahead of time. FastGLT prunes weights and edges once for all inputs and saves 95–98% of multiply-accumulates. It also documents "structural noise", where early edge removal deletes critical paths, and competitors collapsing beyond 70% graph sparsity on Citeseer. Fixed Aggregation Features precomputes neighbor statistics and matches GNNs on 12 of 14 benchmarks, but fails where 10–15 hops are needed. Sparsity-Aware Communication sends only the adjacency non-zeros that are actually needed across GPUs, for up to 14x speedup. Our gates are input-dependent and leave the graph topology intact. They also add a second source of zeros, in the feature matrix, which that communication scheme could exploit but currently does not.

One constraint comes from Probing GNN Activation Patterns. It finds that attention concentrates very large activations on bottleneck edges, and that pruning them raises error by 22–27%. Our gate removes only small activations and never touches attention scores. We will still check that the gated model leaves those large activations intact.

Research Program

The work has three stages.

The first stage builds the substrate. We pretrain two ReLU encoders per graph with self-supervision on the unlabeled graph, as in the Virtual Node Tuning setup: a GraphSAGE-style message-passing network and a Graph Transformer whose MLP blocks use ReLU instead of GELU. We then report the natural activation sparsity per layer, per channel, and by node degree. If pretrained GNN activations turn out to be dense, which is plausible given the smoothing effect of mean aggregation, the budget term has more work to do. That outcome is itself a finding about whether the lazy-neuron effect transfers to message passing.

The second stage is threshold tuning on few-shot episodes. The thresholds start at zero, which reproduces plain ReLU. The budget $\rho$ is swept from natural sparsity down to 5%, and $\lambda$ and the learning rates are tuned on held-out validation episodes only. A second learning rate for the thresholds follows the separation used by the materials fine-tuning paper.

The third stage is efficiency measurement at scale. We apply the same gating to a fully supervised encoder on ogbn-products and ogbn-papers100M, the latter with over 111 million nodes and 1.6 billion edges, and run layer-wise full-graph inference with hidden states stored in a compressed sparse format. We measure peak memory, aggregation multiply-accumulates, wall-clock time with an off-the-shelf sparse-times-sparse kernel, and the accuracy cost.

For the distributed setting we do not build a system. Instead we compute the communication volume that a partitioned 1.5D aggregation would need if feature non-zeros were exchanged instead of dense rows, on the Protein and Reddit partitions used by the Sparsity-Aware Communication work.

Baselines

The naive reference point is the frozen pre-trained encoder with a logistic-regression head, dense and ungated. It fixes the accuracy floor and the cost ceiling.

The strongest existing method is Virtual Node Tuning Virtual Node Tuning. It reports 76.68% accuracy on 5-way 1-shot CoraFull, against 73.38% for the best meta-learning baseline. We will run it twice. The first run uses its published protocol and number. The second run places it on our own ReLU-pretrained Graph Transformer, with the same episodes, the same number of trainable parameters (matching the virtual-node count to the threshold count), and the same hyperparameter search budget. This separates any backbone effect from any adaptation effect.

For 5-shot we also compare against LR-FGDM, which synthesizes nodes with graph diffusion and filters them with a low-rank penalty. It reports moving 5-way 5-shot CoraFull accuracy from about 87.8% to over 90%. It is not a transfer method and uses far more compute, so the fair comparison is accuracy at equal labels, with its training cost reported alongside. For few-shot ogbn-arxiv we found no reported result on Lacuna for either method, so we will run both.

For efficiency, the strongest competitor is static pruning with FastGLT FastGLT, applied to the same supervised encoder at matched aggregation cost. We compare accuracy, since the file reports only multiply-accumulate savings, not accuracy at a matched budget.

Two ablations isolate the key ingredient. The first computes the budget term on the support set only, instead of the unlabeled graph. If that performs as well, the "data-rich budget" argument is wrong. The second uses a single global threshold tuned on validation, instead of learned per-channel thresholds.

Beating Virtual Node Tuning means a statistically reliable gain over many episodes on CoraFull and at least one other benchmark, at no more trainable parameters. A tie at 15% firing rate would still count as a partial win on cost, provided the efficiency results hold.

Why not just…?

Why not just use bias tuning, i.e. learn only the biases of the frozen encoder? Shifting a bias before a ReLU also creates zeros, so this is the most serious threat. The difference is that a truncating threshold removes small responses without shrinking the large ones, whereas a negative bias shifts every value down. With few labels, preserving the magnitude of confident features may matter. This is a hypothesis, not established. We will run bias tuning with the same unlabeled-graph budget at matched parameter count. If it ties, the contribution becomes the budget signal and the efficiency accounting rather than the gate shape, and we will say so.

Why not just threshold a frozen encoder's activations after the fact and train a linear head? This gives most of the efficiency with no adaptation at all. Its weakness is that the zeroed features are chosen with no reference to the task. The single-global-threshold ablation, and the gap between post-hoc and learned per-channel thresholds at equal firing rate, decide whether task-specific selection earns its cost. If the gap is under a point, the honest conclusion is that ReLU graph encoders should simply be deployed sparse, and adaptation is a separate question.

Why not just prompt the frozen backbone with Virtual Node Tuning and prune it separately for speed? Composing the two is a legitimate alternative, and we will run it. Prompting leaves internal activations dense, so pruning must be static, and FastGLT's own analysis shows static edge removal introduces structural noise at high sparsity. The test is accuracy at matched aggregation cost for prompt-plus-FastGLT versus threshold tuning alone.

Why not just use Fixed Aggregation Features, which remove message passing entirely? On shallow, local benchmarks they may well beat everything here on cost. They cannot reuse a pre-trained encoder, however, and their accuracy drops on Roman-Empire, where many hops are needed. We include Roman-Empire so that this trade-off is visible rather than assumed.

Decisive Checks

The few-shot evaluation uses 5-way 1-shot and 5-shot episodes on CoraFull, ogbn-arxiv, Coauthor-CS, Amazon-Computers, and Roman-Empire, following the splits of Virtual Node Tuning and LR-FGDM. We report mean accuracy with 95% intervals over at least 100 test episodes per setting.

The first check is decisive for the adaptation claim. At a 15% firing rate, threshold tuning must reach or exceed Virtual Node Tuning run on the same backbone on CoraFull 1-shot. The unlabeled-graph budget must also beat the support-only budget. If either fails, the regularization story is false.

The second check tests the proposed mechanism. We measure activation overlap between classes on query nodes, in the style of the reinforcement-learning sparsity study. The prediction is that lower overlap tracks higher accuracy across $\rho$. If sparsity rises while accuracy falls, it is acting as capacity removal rather than as a regularizer.

The third check covers efficiency. On ogbn-products the gated supervised encoder should keep test accuracy within one point of its dense version; the file's dense references are 78.50% for GraphSAGE and 79.08% for GraphSAINT from Open Graph Benchmark. It should also cut peak hidden-state memory and aggregation multiply-accumulates by at least 3x.

We will also report measured wall-clock time. If firing concentrates on high-degree nodes, the $\rho|E|d$ estimate will overstate the savings, and the measured edge-weighted firing rate will show that directly. Finally, we will check that the large activations on bottleneck edges survive gating on peptides-func, so we know the gate is not removing the signals that the probing study showed to be essential.