Graph neural networks blur nodes together when they average neighbor features too many times. This failure, called over-smoothing, is usually framed as a problem of stacking too many layers on a fixed graph. Streaming temporal graph models face a version of the same problem that nobody has measured. These are models that keep a running memory vector per node and update it at every new interaction, such as a transaction, an edit, or a reply. Each update mixes a node's memory with its partner's, so a node that has interacted a thousand times has gone through something like a thousand rounds of message passing, over a graph that changes at every step. We propose to test whether these node memories collapse toward a low-dimensional subspace as the stream goes on. We also ask whether this collapse, rather than memorization alone, explains why deep temporal models lose to trivial heuristics on parts of the Temporal Graph Benchmark. Finally, we test whether a cheap high-pass correction to the memory update repairs it.
In plain terms, depth in a streaming model is measured in events, not layers, and it is unbounded. If smoothing happens along physical time, the cure is not deeper or shallower networks. It is a memory update that keeps what makes each node different. A positive answer would bring the over-smoothing toolkit into dynamic graphs. A negative answer would also be informative, because it would rule out representation collapse as the cause of the benchmark failures and point the field back at memorization.
Core Claim
The two research directions have so far been kept apart. On the over-smoothing side, the continuous-time work treats time as computational depth over a frozen graph. GRAND recasts layers as steps of a diffusion equation and stays stable at 32 layers. PH-DGN uses energy-conserving Hamiltonian dynamics to stay stable across 1,000 layers. Both rely on a fixed connectivity structure, and the PH-DGN paper says so as a limitation. On the dynamic-graph side, TGB shows that simple heuristics often beat deep temporal models. TGB also finds that models degrade when many test edges were never seen in training, a quantity it calls the "surprise index." Every evaluation there uses ranking metrics, and none inspects the representations.
Our claim is that a memory-based temporal graph model applies a long product of changing averaging operators to node states. This is the same structure as deep message passing, with one difference: the operator changes at each step. The averaging intuition suggests that such products still drive connected nodes toward agreement, as long as the changing graphs keep linking nodes together over time.
The gated, nonlinear memory updates used in practice may or may not resist this pull, and that is exactly what is unknown. We hypothesize three things:
- The effective rank of the memory matrix of active nodes declines over the stream.
- This decline predicts per-window drops in link-ranking quality after controlling for the window's surprise index.
- Adding a high-pass "diversification" term to the memory update, borrowed from filterbank GNNs, restores both rank and accuracy more reliably than normalization or memory resets.
Research Program
The measurement half reuses a method from static graphs. Are we measuring oversmoothing correctly? shows that the classical energy-based smoothness measure can stay flat while accuracy collapses. That paper argues for continuous rank measures instead, Numerical Rank and Effective Rank, and reports an average Pearson correlation of 0.91 between Effective Rank and accuracy. This result constrains our design. We will not use energy-style smoothness as evidence, partly because normalization layers inflate it artificially.
The setup works as follows. At regular checkpoints of physical time $\tau$, we stack the memory vectors of nodes active in a trailing window into a matrix $M_\tau$ and compute
$$\mathrm{Erank}(M_\tau) = \exp\Big(-\sum_i p_i \log p_i\Big), \quad p_i = \sigma_i / \textstyle\sum_j \sigma_j,$$
where $\sigma_i$ are the singular values of $M_\tau$. We will also track Numerical Rank and the mean pairwise cosine similarity among candidate destination nodes.
The same paper warns that high rank is necessary but not sufficient, since random noise has maximal rank. For that reason we pair every rank curve with a linear probe of node identity, specifically whether a memory vector predicts the node's future partners. This lets us tell "diverse and informative" apart from "diverse and noisy."
The models we study are the memory-based and attention-based temporal models already run in TGB, namely TGN and its relatives, together with the official TGB pipeline. We expect most of the action to be in the forward pass, for two reasons. Training typically truncates gradients through memory across batches, and the per-event computation is shallow. That makes the gradient-collapse mechanism of Backward Oversmoothing less directly applicable here. We will still log gradient norms with respect to memory to check this assumption rather than assume it.
The intervention half borrows its mechanism from FB-GNN. FB-GNN observes that averaging keeps only the low-frequency half of a graph signal. It adds a learnable high-pass channel, which subtracts the neighborhood mean from the node itself, and this preserves node identity on heterophilic graphs. We transplant that idea from the depth axis to the event axis. On each interaction, the memory update receives an extra input: the difference between the node's memory and a decayed running mean of its recent partners' memories. A learnable scalar $\alpha_H$ scales this input.
Compared with FB-GNN, this differs in where the filtering acts, which is along the interaction history rather than across layers. It also needs no Laplacian, so it works naturally when edges arrive, and when they are deleted, by adjusting the running mean. The change adds a few parameters and constant per-event cost. This matters because TGB documents that deep temporal models are already one to two orders of magnitude slower than heuristics at inference.
Baselines
The naive reference points are the non-learning heuristics from TGB: EdgeBank and Persistence for link prediction, and Moving Average for node affinity. They are the strongest practical competitors because they cannot over-smooth at all. If a collapse story is real, it should explain exactly the gap between them and the deep models.
On tgbn-reddit, TGB reports Moving Average at NDCG@10 of 0.559 against TGN at 0.315. Beating Moving Average there is the bar. Closing a large fraction of that gap while improving memory rank would support the claim. Improving rank without moving NDCG would count against it.
The strongest learned baseline on link prediction is NAT, which TGB reports at an MRR of 0.749 on tgbl-wiki. We have no reported result on Lacuna for TGN's own MRR on tgbl-wiki. On the high-surprise tgbl-review dataset, TGB reports that NAT drops and that GraphMixer and TGAT perform best, but we found no exact reported number on Lacuna for these models on tgbl-review. We will therefore rerun them ourselves under the official protocol.
We also include FreeDyG as the strongest frequency-domain model. FreeDyG applies Fourier transforms to interaction sequences, not to the graph. It is the closest existing use of "frequency" in dynamic graphs, and it is reported to rank first on most transductive datasets, though the file gives no per-dataset numbers.
Fairness rules apply to every comparison. Each model uses the same TGB splits, the same historical and random negative samples, the same embedding dimension, and an equal hyperparameter search budget on the validation split. The intervention is applied to TGN only, so that any gain is attributable to the update rule.
The key ablation is the same modified model with $\alpha_H$ fixed to zero. This keeps parameter count and running-mean bookkeeping identical and isolates the high-pass term. A second ablation feeds the running mean without subtraction, to rule out "more input features" as the explanation.
Why not just…?
Why not just use the heuristics, since they already win? On low-surprise datasets they likely will keep winning, and we will report that honestly. TGB itself shows that memorization heuristics fail on high-surprise data such as tgbl-review, where an inductive representation is required. The interesting question is why deep models also fail there. If rank collapse is the reason, the fix should help most exactly where heuristics are weakest. We will test this by stratifying gains by window-level surprise index.
Why not just add LayerNorm or PairNorm to the memory? This is a real threat. Are we measuring oversmoothing correctly? shows that normalization raises energy-style smoothness while the underlying rank still collapses. That result predicts normalization will fail on our rank measure, but it was established on static graphs only. We will run LayerNorm and PairNorm on the memory as direct competitors. If either matches the high-pass term on both rank and MRR, we will drop the architectural claim and keep the diagnostic one.
Why not just periodically reset or decay node memories? Resetting removes accumulated smoothing, but it also discards long-range history, which is what memory is for. We will sweep reset intervals and exponential decay rates under the same tuning budget. If some reset schedule matches our method, that would itself confirm the collapse hypothesis, since resets only help if accumulated mixing is harmful. We would then report the cheaper fix as the recommendation.
Why not just use an energy-conserving model like PH-DGN? Its conservation guarantee depends on a fixed graph, as that paper states. Extending it to switching topologies is open work, and the switching port-Hamiltonian results in Learning Switching Port-Hamiltonian Systems have not been carried to graphs. We see this as a follow-up that our measurements would motivate, not a drop-in baseline.
Decisive Checks
The first check is descriptive. On tgbl-wiki, tgbl-review, tgbl-comment, tgbn-reddit and tgbn-trade, we plot memory Effective Rank and Numerical Rank over the test stream for each trained model. As a control, we use the same models with memory frozen at the end of training.
The second check is explanatory. We regress per-window MRR (or NDCG@10) on Effective Rank and surprise index jointly. The hypothesis survives only if rank carries predictive weight beyond surprise. As a second test, nodes with more interactions should show higher similarity to other heavily active nodes.
The third check is interventional. The high-pass memory update must beat the $\alpha_H = 0$ ablation, the best normalization variant, and the best reset schedule on MRR and NDCG@10. The gains must concentrate in high-surprise windows, inference cost must stay within a small constant of TGN, and the identity probe must improve alongside rank.
If memory rank does not decline, or declines without predicting accuracy, we will conclude that temporal models in TGB fail for memorization reasons rather than smoothing. We will publish the measurement protocol as the contribution.
References
- GRAND: Graph Neural Diffusion
- Port-Hamiltonian Architectural Bias for Long-Range Propagation in Deep Graph Networks
- Temporal Graph Benchmark for Machine Learning on Temporal Graphs
- Are we measuring oversmoothing in graph neural networks correctly?
- Backward Oversmoothing: why is it hard to train deep Graph Neural Networks?
- Complete the Missing Half: Augmenting Aggregation Filtering with Diversification for Graph Convolutional Networks
- FreeDyG: Frequency Enhanced Continuous-Time Dynamic Graph Model for Link Prediction
- Learning Switching Port-Hamiltonian Systems with Uncertainty Quantification