Write Only When Surprised: Surprise-Gated Node Memory for Temporal Graph Networks

Temporal graph networks are models that keep a running memory vector for every node in a stream of timestamped interactions, such as user–item reviews, transactions or co-edits. They spend most of their time and memory bandwidth rewriting node states after events they could have predicted. They are also embarrassingly fragile. On the Temporal Graph Benchmark, a moving-average heuristic beats the Temporal Graph Network (TGN) model at predicting which communities a user will engage with next. It scores 0.559 against 0.315 NDCG@10 on tgbn-reddit, where NDCG@10 is a ranking-quality score over the top 10 predictions. Neural models also run two orders of magnitude slower at inference (Temporal Graph Benchmark for Machine Learning on Temporal Graphs). We propose a different policy. A node's memory is written only when the model was surprised by the event and the surprise looks learnable rather than noise. In all other cases a cheap heuristic state carries the node forward.

The idea borrows from sequence models that store only unpredictable inputs. It also borrows one specific, testable hypothesis from the biology of cellular senescence. There, a stressed cell releases a cocktail of signalling molecules (the senescence-associated secretory phenotype, or SASP) that pushes its immediate neighbours into the same state. Our graph analogue is that a surprising event at one node predicts surprising events at that node's neighbours soon after. If the hypothesis holds, a surprise should also trigger a bounded refresh of the neighbours' memories. The payoff we are testing is threefold: fewer memory writes, larger training batches, and a model that degrades to the heuristic rather than below it when the stream stops resembling the past.

Core Claim

In memory-based temporal graph models, most node-memory writes carry little information. We claim that gating writes on the model's own pre-update surprise, filtered by a learnability test, preserves accuracy with a small fraction of the writes. Neighbour refresh can recover whatever is lost at nodes adjacent to surprising events.

A second, less obvious consequence follows. The scaling problem that PRES attacks is temporal discontinuity: a node touched several times inside one parallel batch gets inconsistent updates. Gating removes most of these same-node collisions before they happen. So surprise gating should allow large temporal batches without PRES's correction filter.

The claim is only interesting if the gate beats random skipping at the same write rate. That comparison is the decisive experiment.

Where the pieces come from

The gating mechanism comes from three lines of sequence-memory work, which supply the method.

  • Adaptive Posterior Learning writes an observation to external memory only when its negative log-likelihood exceeds a threshold. It reports that surprise becomes bimodal during training.
  • Titans uses gradient magnitude as surprise to drive test-time memory updates.
  • Explain My Surprise allocates long-term memory only when local context cannot resolve a transition.

All three operate on a single sequence. None faces the graph-specific issue that one event updates two memories and stales the memories of their neighbours.

The learnability filter is constrained by Information-theoretic Online Memory Selection for Continual Learning. That paper shows pure surprise fills buffers with unlearnable outliers. It fixes this with a Bayesian last-layer information-gain test computed in $O(d^2)$ by rank-one updates. We reuse that computation directly.

On the graph side, two papers motivate the work.

  • TGB measures surprise only as a dataset statistic, the fraction of test edges unseen in training. It shows memorisation-heavy models fall apart when that fraction is high. We turn surprise into an online per-event signal inside the model.
  • PRES keeps every event and corrects the parallel-batch inconsistency with a Gaussian mixture predictor and a coherence loss. We instead remove most colliding writes and ask whether correction is still needed.

SpikeNet is the closest existing sparsification. Its nodes emit binary spikes when accumulated input crosses a threshold. That threshold reacts to the volume of incoming messages, not to prediction error, so a flood of routine events fires it as readily as one decisive event.

The SASP literature plays a narrower role: it supplies a hypothesis, not a mechanism.

Two features transfer. First, disruption spreads locally to topological neighbours rather than globally. Second, the literature distinguishes transient secretion, which aids repair, from chronic secretion, which is pathological. In our model, the analogue of chronic secretion is a node that stays surprised event after event without its predictions improving. The learnability filter should route that pattern to the heuristic rather than keep writing it.

We do not claim the biology validates the design. We claim it names a propagation pattern that we can measure in graph streams and discard if absent.

Research Program

We build the gate on TGN's standard predict-then-update loop. For an incoming event $e=(u,v,t)$, TGN scores the link from the current, pre-update memories before writing. The surprise of the event is therefore available at no extra cost:

$$s_e = -\log \sigma\big(f(m_u, m_v, t)\big),$$

Here $f$ is the link decoder, $m_u$ and $m_v$ are the stored memories of the two endpoints, and $\sigma$ is the logistic function.

Following InfoRS, we treat the decoder's final layer as Bayesian linear regression with posterior covariance $\Sigma$ and noise variance $\sigma_n^2$. The information gain of the event's final-layer features $\phi_e$ is

$$g_e = \tfrac{1}{2}\log\big(1 + \phi_e^\top \Sigma\, \phi_e / \sigma_n^2\big).$$

A memory write, meaning the recurrent (GRU) update plus storage of the raw message, is executed only if both $s_e > \tau_s$ and $g_e > \tau_g$. The thresholds are set to hit a target write rate so that comparisons are at matched budget. Non-written nodes keep their stale memory. For node-affinity tasks, a non-written node also carries a moving-average state that the decoder can read.

Neighbour refresh is added only if the contagion measurement below supports it. When $u$ receives a write, its recent 1-hop neighbours are flagged, and each flagged neighbour's next event is written regardless of its own surprise. This mirrors paracrine induction and costs one bit per node.

A clearance rule is the analogue of senolytic drugs, which remove senescent cells. It frees the memory slot of any node whose memory has not been written within a horizon $H$ and whose recent predictions do not beat the heuristic. That node reverts permanently to the heuristic until surprised again. This bounds memory by the number of "interesting" nodes rather than all nodes.

The analytic part of the program comes first and is cheap. On every TGB dataset we log $s_e$ for a fully trained TGN and measure three things:

  • the fraction of events below each surprise quantile;
  • the rate of same-node collisions inside a batch as a function of batch size;
  • the "surprise contagion" statistic, meaning the probability that a neighbour's next event is in the top surprise decile given that its neighbour's last event was, relative to the unconditional rate.

A contagion ratio near 1 kills neighbour refresh and leaves a plain gating paper. A ratio well above 1 justifies it.

Baselines

Naive reference points. These are the TGB heuristics: moving average for node affinity and EdgeBank for link prediction. We run them through the official TGB evaluation. The reference numbers are Moving Average NDCG@10 of 0.559 against TGN's 0.315 on tgbn-reddit (TGB). For link prediction the strongest neural reference in the file is NAT, with MRR 0.749 on tgbl-wiki (mean reciprocal rank of the true edge), from the same paper. On tgbl-review, TGB reports only that GraphMixer and TGAT lead and NAT drops; we found no numeric value on Lacuna. Beating the heuristics on node affinity means matching or exceeding 0.559 on tgbn-reddit while writing a minority of events. A result below the heuristic is a failure even if writes fall sharply.

Full-write TGN. This is the method we modify. It is the fairest accuracy ceiling for the gate and the reference against which write reduction is counted. It is trained with identical hyperparameters, neighbour sampler, seeds and epochs.

PRES. PRES is the strongest existing method for scaling memory-based training. It reports TGN-PRES AP of 98.0% against 98.4% for small-batch TGN on REDDIT (average precision), a 3.4× training speedup, and 4× larger temporal batches. We run PRES and our gate on REDDIT, WIKI, MOOC, LASTFM and GDELT with the same base TGN, the same GPU and the same wall-clock budget, sweeping batch size upward. Beating PRES means sustaining AP within 0.5 points of small-batch TGN at batch sizes beyond 4×, or matching PRES's AP at 4× with lower time per epoch. We also run the combination, gate plus PRES, since the two may be complementary.

Ablations isolating the key ingredient. The most important is random skipping: events dropped uniformly at random at exactly the gate's write rate. It is the graph analogue of reservoir sampling, is information-blind, and is what a skeptic would try first. Other ablations are:

  • surprise-only gating, without the learnability filter, which tests the outlier trap;
  • gating by recency (write if the node has not been written in $k$ events);
  • gating with and without neighbour refresh;
  • clearance compared with a sliding-window, least-recently-used eviction at equal memory.

All ablations share thresholds tuned on validation only, with the same tuning budget of trials per variant.

Why not just…?

Why not just ensemble the moving average with TGN? A learned blend of the two could plausibly recover the heuristic's node-affinity accuracy. On accuracy alone this is a real threat. But an ensemble still performs every memory write and pays TGN's full inference cost, the cost TGB flags as two orders of magnitude above heuristics. Our proposal claims accuracy at a fraction of the compute. We will run the ensemble as a baseline. If it matches our accuracy and random skipping matches our compute, the gate has no reason to exist, and we will say so.

Why not just skip events at random? InfoRS shows that uniform retention discards rare, decisive samples under imbalance, and keeps less accuracy with a 400-point buffer (Information-theoretic Online Memory Selection). That evidence comes from image continual learning, not graphs. So this is settled empirically by the matched-write-rate comparison above, reported across write rates from 50% down to 5%. The gate must dominate the random curve, especially on the high-surprise tgbl-review.

Why not just use PRES with bigger batches? PRES already reaches 4× batches with a small AP cost. Its correction exists because colliding same-node updates are inconsistent. If gating removes most collisions, the correction becomes unnecessary, and the gate additionally cuts per-event compute, which PRES does not. The collision-rate measurement tells us in advance whether this can work. If collisions are dominated by genuinely surprising bursts at hub nodes, gating will not help batching, and the gain reduces to write savings.

Why not just binarize states as SpikeNet does? SpikeNet cuts parameters about 50% relative to TGAT and scales to the 2.7M-node Patent graph (SpikeNet). Its firing rule responds to message volume, so a node that receives many routine events fires as often as one that receives a single decisive event. Binarization and gating are also orthogonal, so we test gating on a spiking backbone as a secondary experiment rather than treating the two as rivals.

Proposed Experiments

The main evaluation uses the official TGB protocol and splits. Link prediction runs on tgbl-wiki, tgbl-review, tgbl-coin and tgbl-comment, reporting MRR. Node affinity runs on tgbn-trade, tgbn-genre, tgbn-reddit and tgbn-token, reporting NDCG@10. For each run we also report:

  • the fraction of events that trigger a memory write;
  • inference throughput in events per second;
  • peak GPU memory;
  • the number of live memory slots after clearance.

Results are stratified by per-event novelty, meaning whether the pair was seen in training, which ties back to TGB's surprise index. Scaling experiments on the PRES datasets report AP against temporal batch size and time per epoch.

Three checks decide the project.

The first is the pre-registered analysis. If more than half of TGN's events are low-surprise and the gate at a 20% write rate stays within one MRR point of full-write TGN on tgbl-wiki, the core claim survives. If accuracy tracks write rate as closely for the gate as for random skipping, it fails.

The second is tgbn-reddit, where a gated model that falls back to the moving average must at least reach the heuristic's 0.559. Otherwise the fallback design is wrong.

The third is the contagion ratio together with the refresh ablation. Neighbour refresh stays in the method only if the ratio is clearly above 1 and refresh improves accuracy at the neighbours of written nodes at equal total writes. This is the single place where the senescence hypothesis is either borne out in data or dropped.