Graph neural networks compute each node's representation by repeatedly averaging information from its neighbors. Repeat this enough times and every node in a connected region ends up with nearly the same vector. This failure is called over-smoothing. In temporal graph models, which process timestamped interactions such as Wikipedia edits, product reviews, or crypto transfers, the repetition does not only come from stacking layers. It also comes from time. Memory-based models such as TGN refresh each node's stored state every time it interacts, mixing in its neighbors' stored states, and those states were themselves mixed earlier. A stream of millions of events therefore acts like a very deep network. Many temporal models also use "temporal decay attention," which shrinks the attention paid to old interactions by a decreasing function of elapsed time. These models look as if they should be protected from that collapse.
We argue that whether decay protects against collapse depends almost entirely on where the decay function is inserted into the attention computation, and that the most common insertion point offers no protection at all. If the decay multiplies the attention score before the softmax normalization, stale neighbors are pushed toward uniform weight, not toward zero. That blurs attention instead of cutting stale links. If instead the logarithm of the decay is added to the score, stale weights actually vanish. Whether representations then collapse depends on whether the total weight given to stale information over the stream is finite. We propose to prove this distinction, measure it as the loss of rank of the node-memory matrix over stream time on the Temporal Graph Benchmark, and derive one design rule from it. The rule is additive decay measured on each node's own clock, which is meant to fix the known failure of decay on sparse, infrequently active nodes.
Core Claim
Write the attention a node $i$ pays to a temporal neighbor $j$ as $\alpha_{ij} = \exp(s_{ij}) / \sum_k \exp(s_{ik})$. Here $s_{ij}$ is the attention score, $q_i \cdot k_j$ is the query–key similarity, and $\psi(\Delta t) \in (0,1]$ is a decay kernel of elapsed time $\Delta t$. The research file records two standard ways to combine them:
$$s^{\text{mult}}_{ij} = q_i \cdot k_j \,\psi(\Delta t_{ij}), \qquad s^{\text{add}}_{ij} = q_i \cdot k_j + \log \psi(\Delta t_{ij}).$$
Under the multiplicative form, as $\Delta t \to \infty$ the score goes to zero. The weight therefore goes to $1/Z$, the same weight every content-free neighbor receives. Stale edges lose their discriminating content but keep their full share of the averaging. This is exactly the strictly positive, connectivity-preserving attention that Demystifying Oversmoothing in Attention-Based Graph Neural Networks proves cannot prevent exponential collapse to identical rows. If anything, multiplicative decay should speed collapse, because it flattens the scores.
Under the additive form, $\alpha_{ij} \propto \psi(\Delta t_{ij}) \exp(q_i \cdot k_j)$, so stale weights really do go to zero. That result's analysis assumes the weights stay bounded away from zero, which no longer holds here, so its collapse result does not automatically apply. Our working hypothesis follows the classical intuition for products of stochastic matrices. Collapse to consensus is avoided only when the cross-group mixing a node absorbs over the whole stream sums to a finite amount. Whether that happens is a joint property of the kernel and of the event rate. The same global $\exp(-\lambda \Delta t)$ can therefore protect a frequently active node and starve a rarely active one.
This last point lines up the theory with the sparsity failure reported for kernel decay in Kernelized Edge Attention: Addressing Semantic Attention Blurring in Temporal Graph Neural Networks. The design that falls out is additive decay with $\Delta t$ divided by the node's own typical gap between events, $\tau_i$. Time is then counted in "how many of my own interactions ago," not in seconds.
Relation to Prior Work
Several papers motivate the question. The static-graph collapse proof above supplies the theory we extend. It covers time-invariant topologies only and never considers weights that vanish over time. Are we measuring oversmoothing in graph neural networks correctly? shows that a common measure of how much neighboring vectors differ, Dirichlet energy, can look healthy while accuracy collapses. It shows that the effective rank of the feature matrix (Erank, the entropy of its normalized singular values) tracks accuracy with an average Pearson correlation of 0.91. We take Erank and numerical rank as our primary collapse measures. We also heed that paper's warning that high rank is necessary but not sufficient, since random noise has maximal rank.
Three temporal-decay papers supply the mechanisms we compare. KEAT modulates edge representations with Laplacian or RBF kernels to stop "semantic attention blurring" in one- or two-layer models. Time-Aware and Transition-Semantic Graph Neural Networks for Interpretable Predictive Business Process Monitoring injects a global $\exp(-\lambda\delta)$ into graph attention to stop "signal drowning." TSGAN applies exponential decay to user timelines. None of the three measures rank across depth or stream time, and none distinguishes the placements above. Our contribution is to say which of their mechanisms can, in principle, stop collapse, and why.
Two papers constrain the design. A Dynamical Systems-Inspired Pruning Strategy for Addressing Oversmoothing in Graph Attention Networks (DYNAMO-GAT) cuts edges between highly correlated nodes and survives 128 layers. It warns that this can disconnect sparse graphs, which is a hazard our node-local clock is designed to avoid. A Signed Graph Approach to Understanding and Mitigating Oversmoothing shows that residual connections and normalization act like weak negative edges and only delay collapse. Setting the Record Straight on Transformer Oversmoothing cautions that some smoothing is beneficial. For that reason we judge every method by task metrics alongside rank, never by rank alone.
Finally, Temporal Graph Benchmark for Machine Learning on Temporal Graphs supplies the evaluation. It also supplies a provocation: a moving average beats TGN on node-affinity tasks. Our framing predicts this could be partly a collapse effect, because a moving average has no cross-node mixing to collapse.
Research Program
The theoretical part extends the matrix-product analysis to time-indexed attention matrices whose entries may vanish. The goal is to prove three things. First, multiplicative-in-score decay preserves the positivity lower bound and so inherits exponential collapse. Second, additive decay escapes collapse when, roughly, the stale cross-group weight summed over each node's event sequence converges. Third, measuring time on a node-local clock makes that sum depend on the number of the node's own interactions rather than on wall-clock gaps. That removes the regime where rarely active nodes lose all context.
The empirical part instruments memory-based and attention-based temporal models so that Erank and numerical rank of the full node-state matrix are logged at regular points along the stream. The same rank measures are also logged as attention layers are stacked from 1 to 32. A synthetic temporal stochastic block model is the controlled testbed. It has known communities, adjustable event rates per community, and a tunable fraction of rarely active nodes. On it, collapse toward identical community vectors can be checked against ground truth, not just against rank.
Baselines
The naive reference points are the heuristics from the benchmark paper. They are the moving average on node-affinity tasks, which reached NDCG@10 of 0.559 on tgbn-reddit against TGN's 0.315, and EdgeBank on link prediction. Both have no mixing and therefore no collapse. Any learned method that loses to them while showing falling rank supports our diagnosis, not our method. They are run with the official TGB evaluation protocol and require no tuning.
The strongest existing decay method is KEAT applied to TGN and DyGFormer backbones, since it is the only kernel-decay approach evaluated on TGB. The file reports only relative gains for it, 18.8% MRR on tgbl-review, 7.7% on tgbl-wiki and 6.9% NDCG@10 on tgbn-trade, with no absolute KEAT score found on Lacuna. We will therefore rerun it ourselves on identical splits, negative samples, and seeds. KEAT's kernel bandwidth and our decay rate get the same tuning budget, a fixed-size search on the validation stream. For absolute link prediction context, the strongest number in the file is NAT's MRR of 0.749 on tgbl-wiki. To count as beating KEAT, our variant must match or exceed its task metric on every dataset. It must also clearly beat KEAT on the rarely active node subgroup, where its known failure lies, and keep higher memory-matrix Erank late in the stream.
The adapted pruning baseline transfers DYNAMO-GAT's correlation-based edge removal to the temporal neighbor sampler under the same compute budget. It tests whether hard cutting by feature correlation can do the same job as soft cutting by time.
The key ablation holds the kernel, backbone, and tuning fixed and varies only the placement. The placements are multiplicative in the score, additive log-decay, and KEAT-style edge-feature modulation, each crossed with a global clock versus a node-local clock. The theory predicts a specific ordering of these six cells in late-stream rank, and that ordering is the paper's central result.
Why not just…?
Why not just add residual connections or normalization to the memory updater? TGN's GRU memory already has a gate that retains part of the old state, which is itself a form of decay. This is a real threat to our claim. The signed-graph analysis and the rank-measurement paper both find that such fixes delay collapse but do not stop it. Our experiments include GRU, GRU with LayerNorm, and residual variants as backbones. If any of them holds late-stream Erank flat and matches additive decay on task metrics, the placement result loses most of its practical force and we will report it that way.
Why not just use the most-recent-k neighbor window that temporal models already sample? A hard window cuts old edges at every step, but memory carries history forward, so each step still mixes. A window bounds how far back each step looks, not the total mixing accumulated over the stream. The synthetic block model tests this directly by sweeping k at fixed decay. If small windows alone prevent collapse, the additive-decay design is redundant.
Why not just feed time encodings into the features and let attention learn recency? KEAT already tested this. It found that summed time encodings get swamped by slow-changing node states and produce blurred attention, while explicit kernel modulation gave large MRR gains. We keep time encodings in every variant, so any gain we report comes on top of them.
Why not just tune one global decay rate per dataset? The process-monitoring paper does exactly this. Its own results show decay losing to simpler models on the sparse BPI20 log, consistent with a global rate starving infrequent cases. The global-versus-local cell of the ablation measures how much a node-local clock adds beyond a well-tuned global rate.
Proposed Experiments
We run on tgbl-wiki, tgbl-review, tgbn-trade, tgbn-reddit and tgbn-genre from TGB, plus the JODIE datasets. Metrics are MRR and NDCG@10. All results are reported separately for node subgroups split by activity frequency, alongside Erank and numerical-rank trajectories over stream time and over depth.
The first decisive check is whether vanilla TGN's memory matrix loses rank over the stream, and whether that loss tracks NDCG@10 against the moving-average baseline. If rank does not fall, the stream-as-depth premise fails. The project then narrows to the layer-stacking question, which we test on DyGFormer and KEAT-TGN at 1 to 32 layers.
The second check is the placement ordering. Multiplicative decay should collapse at least as fast as no decay. Additive decay should preserve rank. The node-local clock should remove the rarely active subgroup's loss.
The third check guards against rank inflated by noise. Any rank gain must come with a task-metric gain and with recoverable community structure on the synthetic graph.
References
- Demystifying Oversmoothing in Attention-Based Graph Neural Networks
- Kernelized Edge Attention: Addressing Semantic Attention Blurring in Temporal Graph Neural Networks
- Are we measuring oversmoothing in graph neural networks correctly?
- Time-aware and transition-semantic graph neural networks for interpretable predictive business process monitoring
- TSGAN: Temporal Social Graph Attention Network for Aggressive Behavior Forecasting
- A Dynamical Systems-Inspired Pruning Strategy for Addressing Oversmoothing in Graph Attention Networks
- A Signed Graph Approach to Understanding and Mitigating Oversmoothing
- Setting the Record Straight on Transformer Oversmoothing
- Temporal Graph Benchmark for Machine Learning on Temporal Graphs