Graph neural networks (GNNs), models that compute each node's representation by repeatedly averaging information from its neighbors, are known to "over-smooth": stack enough averaging steps and every node ends up looking the same. The usual reply is that real models are shallow, so the problem does not arise. Temporal graph models that keep a per-node memory are a different case. Every time two nodes interact, each node's memory is updated using the other's memory. Over a stream of millions of events, that is millions of mixing steps, even though the network has only one or two layers. We propose to test whether memory-based temporal GNNs over-smooth across time rather than across layers. If they do, it may explain a puzzling benchmark result. On the Temporal Graph Benchmark, a per-node moving average, which never mixes information between nodes, beats TGN on node affinity prediction (NDCG@10 of 0.559 versus 0.315 on tgbn-reddit, Temporal Graph Benchmark for Machine Learning on Temporal Graphs).
The proposal has three parts. The first is a measurement: track how many effective dimensions the matrix of all node memories occupies as the stream advances. The second is a small theory that treats memory updates as a product of time-varying mixing matrices. The third is a targeted fix: anchor each node's memory to a decaying summary of its own recent observations. This keeps nodes distinct without tying them to stale initial features, and it gives the recurrence a finite memory horizon that a stability-style generalization argument can use.
Core Claim
Two hypotheses are at stake. First, in memory-based temporal GNNs, the representational rank of the node-memory matrix collapses as a function of the number of processed events. Second, the size of this collapse predicts how far the model falls behind per-node heuristics. Here "representational rank" means the effective number of independent directions the node vectors span, measured by Effective Rank and Numerical Rank.
A source term inside the memory update can prevent the collapse. That term pulls each node toward an exponentially decayed average of its own past messages. Placing it inside the recurrence should outperform both of the obvious alternatives: anchoring to initial features, and feeding the moving average in as an extra input at prediction time.
What would be new is the choice of axis. Rank collapse has been measured and countered only over layers on static graphs. Here it is measured over events in a stream, and the anchor is designed to drift with the node rather than stay fixed.
Grounding in Prior Work
Several papers motivate the diagnostic. Are we measuring oversmoothing in graph neural networks correctly? shows that Dirichlet energy, the standard measure of how much neighboring features differ, can stay stable while accuracy drops by layer 10 in trained GCNs. Effective Rank, by contrast, tracks the drop. We adopt its rank metrics unchanged but apply them to memory snapshots indexed by stream time, not by layer. That paper also warns that high rank is necessary but not sufficient, since random noise has high rank. We therefore never report rank without task accuracy. Preventing Representational Rank Collapse in MPNNs by Splitting the Computational Graph counters collapse by splitting edges according to degree orderings. In a stream those orderings shift constantly, so we do not reuse its design.
Other papers supply the mechanism of the fix. Why Do Attributes Propagate in Graph Convolutional Neural Networks? shows that initial residual connections are exactly the gradient step of a term keeping representations close to the inputs. Graph Neural Networks Inspired by Classical Iterative Algorithms builds layers as optimization steps with a proximal anchor to the input features. DIGNN (Implicit Graph Neural Diffusion Networks: Convergence, Generalization, and Over-Smoothing) proves that over-smoothing at inference time occurs precisely when diffusion lacks such a source constraint. It also links fast convergence of the propagation process to better generalization.
All three papers anchor to a fixed $X$ on a fixed graph. That is the tension the research file flags: a fixed anchor goes stale under drift. We keep their structure (a smoothing term plus a source term) but replace the fixed anchor with a moving, per-node one. Anti-Symmetric DGN offers a different route, non-dissipative dynamics, and names dynamic graphs as its future work. We include it as a competing fix rather than building on it. One reason is that energy conservation preserves stale information just as faithfully as fresh information.
The application setting is constrained by Temporal Graph Benchmark for Machine Learning on Temporal Graphs. It reports that heuristics win on several node-affinity tasks, while structural learners such as TGAT and GraphMixer win on high-"surprise" datasets like tgbl-review, where many test interactions were never seen before. Any credible fix must keep the second advantage while closing the first gap.
Research Program
Measurement. We instrument TGN-style memory models and record the full memory matrix $M(t) \in \mathbb{R}^{n \times d}$ at regular event checkpoints, where $n$ is the number of active nodes and $d$ is the memory width. We compute Effective Rank and Numerical Rank of the centered $M(t)$, both globally and within interaction communities.
This is done on all TGB node and link datasets in the file. For contrast, we run memoryless models (GraphMixer, TGAT), whose representations are recomputed from a fixed window and so cannot accumulate mixing. The prediction is that rank declines with event count for memory models and not for memoryless ones. A second prediction is that the decline is steeper on datasets where the moving average wins.
Theory. We linearize a memory update as $m_v \leftarrow (1-\alpha) m_v + \alpha m_u$ for an interaction $(u,v)$, where $\alpha$ is the effective mixing weight. The stream is then a product $M(t) = P_t \cdots P_1 M(0)$ of row-stochastic matrices, each touching two rows. When the interaction graph is jointly connected over windows, such products contract the memories toward a rank-one subspace. The contraction rate is governed by how fast information spreads through the accumulated interaction graph.
The anchored update replaces this with
$$m_v \leftarrow (1-\alpha-\beta)\, m_v + \alpha\, m_u + \beta\, a_v(t), \qquad a_v(t) = \gamma\, a_v(t^-) + (1-\gamma)\, x_v(t),$$
Here $\beta$ is the anchor strength, $a_v(t)$ is node $v$'s exponentially decayed average of its own incoming message features $x_v(t)$, $t^-$ denotes the moment just before the current event, and $\gamma$ sets the anchor's decay.
We aim to prove two results. The first is a lower bound on the spread of $M(t)$ in terms of $\tfrac{\beta}{\alpha+\beta}$ times the spread of the anchor matrix $A(t)$. This is a streaming analogue of DIGNN's source-constraint result. The second is that the recurrence becomes contractive with rate $1-\beta$, so the influence of events older than roughly $1/\beta$ steps decays geometrically. That finite horizon is what lets a stability argument bound the gap between training and test error under a slowly drifting stream: perturbing one old event moves the current memory by an amount shrinking like $(1-\beta)^{\text{age}}$.
The theory is linear, as in DIGNN and GCC. We will state plainly that it covers the GRU update only through linearization.
Method. The anchor is added inside the memory update of TGN and of one other memory model. $\beta$ and $\gamma$ are learned per dataset, and the anchor is computed in the same streaming pass, so it costs one extra vector per node.
Baselines
The naive reference points are the TGB heuristics: Moving Average and persistence for node affinity, and EdgeBank for link prediction. They need no training, cannot over-smooth, and set the bar the file says neural models often miss. The fix beats them on tgbn-reddit if it exceeds the reported NDCG@10 of 0.559 (Temporal Graph Benchmark). The unmodified TGN, at its reported 0.315, is the model we modify, so closing most of that gap is the first success criterion.
For link prediction, the strongest reported competitor is NAT, with a best MRR of 0.749 on tgbl-wiki (Temporal Graph Benchmark). On tgbl-review, GraphMixer and TGAT outperform NAT, but no reported result was found on Lacuna for their exact scores, so we will reproduce them under the TGB protocol. Matching NAT on tgbl-wiki while keeping the structural learners' advantage on tgbl-review would count as success. Large gains on the heuristic-favoring datasets alone would not.
A-DGN, adapted to streaming by applying its anti-symmetric update to memories, is the strongest alternative anti-collapse mechanism. Beating it would show that a drifting source term is the right remedy rather than energy conservation.
The ablations isolate the key ingredient. We set $\beta=0$; anchor to initial node features instead of $a_v(t)$, the static-anchoring design of GCC and TWIRLS; and supply $a_v(t)$ only as an extra input to the prediction head (late fusion). Every model uses the TGB splits, historical negative sampling, the same memory width, and an equal hyperparameter search budget measured in trials and GPU hours, with five seeds.
Why not just…?
Why not just ensemble TGN with the moving-average heuristic? This is the most serious threat, because late fusion supplies the same per-node signal at almost no cost. If late fusion matches the anchored model, then the gap to heuristics is about a missing feature, not about collapse inside the recurrence, and the core claim fails. The decisive comparison is the late-fusion ablation at matched parameters, combined with the rank traces. Our hypothesis predicts that late fusion leaves memory rank collapsed and underperforms on tgbl-review-style datasets. There, the useful information lives in multi-hop memory that collapse has already erased.
Why not just add LayerNorm or a reset to the memory? Are we measuring oversmoothing in graph neural networks correctly? found that normalization layers mask the energy metrics without stopping rank collapse. Periodic resets discard exactly the long history that memory models exist to keep. We include both as ablations and report their rank traces alongside their accuracy.
Why not just use memoryless temporal models? On tgbl-review they already win, which shows that recomputation from a window avoids accumulation. But they also lose per-node long-term state, and they do not explain why memory models fail. If memoryless models dominate everywhere once tuned equally, the right conclusion is to abandon memory rather than repair it, and we would report that.
Decisive Checks
The project stands or falls on four outcomes. First, Effective Rank of TGN memory declines measurably with event count on at least the node-affinity datasets, and stays flat for GraphMixer. Second, across datasets, the size of that decline correlates with TGN's deficit relative to the moving average. Third, the anchored model stops the decline and closes most of the 0.315-to-0.559 gap on tgbn-reddit without losing ground on tgbl-review. Fourth, the inside-the-recurrence anchor beats both static anchoring and late fusion. Static anchoring should degrade as streams lengthen and drift, which would confirm the stale-anchor tension directly.
As a control for the "high rank is not enough" caveat, we inject noise into memories so that rank rises artificially. If accuracy does not improve, rank is confirmed as a necessary rather than a sufficient diagnostic. If the first check fails and memory rank does not collapse over time, the hypothesis is refuted cleanly, and the measurement study is still a useful negative result.
References
- Temporal Graph Benchmark for Machine Learning on Temporal Graphs
- Are we measuring oversmoothing in graph neural networks correctly?
- Preventing Representational Rank Collapse in MPNNs by Splitting the Computational Graph
- Why Do Attributes Propagate in Graph Convolutional Neural Networks?
- Graph Neural Networks Inspired by Classical Iterative Algorithms
- Implicit Graph Neural Diffusion Networks: Convergence, Generalization, and Over-Smoothing
- Anti-Symmetric DGN: a stable architecture for Deep Graph Networks