Cascade-Closed Windows for Communication-Light Training on Unraveling Networks

When users leave an online community, they often leave in chains: one departure drops a neighbor below the level of engagement that kept them around, that neighbor leaves, and the loss spreads. Predicting these chains requires models that read a long stream of interactions in time order, keeping a running "memory" per user. That ordering requirement makes the models hard to train on many machines. Today's systems give each machine a contiguous slice of time. Under infrequent synchronization, a machine that happens to see a stable period and a machine that sees a collapse learn very different weights, and averaging those weights fails.

This note proposes cutting the stream into training windows only at moments when no departure chain is in progress, and giving each window a short warm-up prefix long enough to rebuild user memory. If those cuts are chosen well, the windows become roughly interchangeable samples. They can then be shuffled freely across machines, and machines can train for many local steps between weight averages without drifting apart. The quantity of interest is whether this lets attrition models train over slow interconnects at near the accuracy of fully synchronized, strictly chronological training.

Core Claim

Sliding-window cropping is usually treated as data augmentation for small datasets. Here it plays a second role: it is the device that turns a sequential stream into samples a distributed optimizer can treat as independent. The danger is that arbitrary crops sever exactly the long, multi-hop chains that unraveling research cares about. Finding Critical Users for Social Network Engagement: The Collapsed k-Core Problem shows that on DBLP at $k=20$ a single author's removal pushed 74 researchers out of the core, although only 12 of them were direct neighbors.

We therefore define cut points using the combinatorial structure of the collapse itself. A $k$-core is the largest subgraph in which every user keeps at least $k$ active neighbors. A cut is "cascade-closed" at time $t$ if no peeling chain of this core, meaning a sequence of departures in which each one pushes a boundary user below $k$, began within the preceding interval $\Delta$ and remains unfinished at $t$. The warm-up prefix length is set from the measured temporal reach of chains.

The claim is that cascade-closed windows have two effects. First, they preserve the accuracy of full-stream memory training on cascade-driven departures, where arbitrary crops lose it. Second, because they make windows exchangeable across workers, local-update training of the DiLoCo type remains stable at synchronization intervals where contiguous time partitioning diverges. The work stands or falls on the conjunction of the two effects: interchangeable windows that do not cut chains.

Relation to Prior Work

Two systems already pair temporal slicing with dynamic graphs. NeutronStream trains dynamic graph networks with an adaptive sliding window that grows when incoming events touch nodes already active in the window. It reports an average 3.97% AUC gain over static batching. This motivates the proposal, since it shows that window boundaries matter. It also differs from it in three ways: its expansion rule is based on immediate co-occurrence rather than multi-hop collapse, it runs on one shared-memory machine, and it does not address synchronization.

DistTGL is the closest work and the main competitor. It scales memory-based temporal graph networks across GPUs, in part through "memory parallelism," which assigns each trainer group a contiguous time segment with its own node memory. It reports a 10.17x throughput speedup over single-machine TGL on GDELT and a 14.5% accuracy improvement there. DistTGL keeps per-step gradient synchronization and contiguous segments. We replace both, with shuffled cascade-closed windows and infrequent averaging, and evaluate on attrition rather than only link prediction.

The cut-point machinery comes from the unraveling literature. CKC shows that only users whose degree is exactly $k$ inside the core, together with their neighbors, can set off follower collapses. Incremental Graph Computation: Anchored Vertex Tracking in Dynamic Social Networks maintains a removal order incrementally as edges change. Together they let us track active peeling chains cheaply over a stream. IncAVT also constrains the design: its maintenance degrades once more than about 17% of edges change in a batch. Our detector must therefore fall back to recomputation during bursts. Such bursts are common, as the bursty contraction episodes in The bursty dynamics of the Twitter information network show.

Peer Influence on Attrition in Massively Open Online Courses supplies both the empirical target and the naive reference point. Its survival analysis finds that a peer dropping out in the same week raises dropout hazard 5.05-fold and a peer dropping out in the prior week raises it by 62%. This suggests that contagion has a reach of one to two weeks, which is a prior for the warm-up length.

The synchronization side reuses DiLoCo. In DiLoCo, workers take $H$ local steps (typically 30), and the averaged weight change is then applied with an outer Nesterov optimizer. It was validated only on text shards that are close to identically distributed. WASH and MERIT both document that independently trained workers drift into incompatible basins when their data differ. That drift is the failure we expect contiguous temporal partitions to trigger. WASH counters drift by swapping parameters between workers; MERIT counters it by splitting conflicting tasks before merging. We instead attack drift at the data-assignment level.

SAMLA predicts guild health from multi-relational graph encoders plus a temporal Transformer, with churn as an auxiliary task. Its fixed observation windows and centralized training are exactly what we vary.

Research Program

We first build the cascade detector. Interaction streams are turned into a time-decayed activity graph: a user counts as active while they have an interaction within a horizon $\tau$, and departure is defined as sustained inactivity. An IncAVT-style removal order is maintained over this graph, and every peeling chain is logged with its start time, duration, and hop depth. This yields an empirical distribution of chain durations, from which we set $\Delta$ and the warm-up length $P$. As a starting rule, $P$ is the 95th percentile of chain duration; it is later swept.

Cascade-closed cut points are the times at which no chain younger than $\Delta$ is unfinished. Training windows are sampled by random cropping. Following the onset-robustness crops of Ghiasi et al., window lengths are drawn from multiple scales. Each window's start is snapped to the nearest cascade-closed cut, and a prefix of length $P$ is prepended. On the prefix, node memory is rebuilt without computing loss.

The model is a memory-based temporal graph network of the kind DistTGL trains, with DistTGL's static-plus-dynamic hybrid memory. It predicts each active user's departure within a fixed future horizon and is evaluated with link prediction as a secondary task.

Distributed training then shuffles windows across $M$ workers. Each worker rebuilds memory locally from its prefixes, so no node memory crosses machines. Workers synchronize weights DiLoCo-style every $H$ steps. The price of this design is recomputation on prefixes, and we measure that cost directly. The analysis tracks worker-weight dispersion before each averaging step and the gap between the averaged model's loss and the mean of the workers' losses. Both are reported as functions of $H$ and of how heterogeneous each worker's windows are in cascade intensity.

Baselines

The naive reference point is the Cox-style survival model of Peer Influence on Attrition in Massively Open Online Courses. It uses time-lagged peer-dropout features and has no neural model and no distribution. It is run on the same departure labels and horizons. On its Coursera forum data it reports hazard ratios (5.05 for same-week peer dropout), not prediction AUC, so we compute its AUC ourselves. Any neural method that fails to beat it on cascade-driven departures has no case.

The upper reference point is single-machine, strictly chronological training of the same temporal graph network over the full stream, with no cropping. This is the accuracy target that distribution must approach.

The strongest existing distributed method is DistTGL, run with its own memory-parallel, epoch-parallel and per-step-synchronized configuration on the same GPU count, model size, data, and hyperparameter search budget. Its reported figures are the 10.17x throughput speedup over single-machine TGL on GDELT and the 14.5% accuracy improvement on GDELT. Lacuna has no reported attrition-prediction result for DistTGL. We additionally run DistTGL's contiguous segments under DiLoCo averaging with the same $H$ values as ours, which isolates the assignment question.

The windowed single-machine baseline is NeutronStream-style co-occurrence-adaptive windowing, which reports an average AUC gain of 3.97%.

The key ablation replaces cascade-closed cuts with uniformly random cuts. It keeps the same warm-up length, window-length distribution, shuffling and $H$, so that only the cut rule differs. A second ablation sets the warm-up length to zero. Beating DistTGL means reaching its attrition AUC within one point, at lower wall-clock time to target under throttled bandwidth (1 and 10 Gbit/s), with $H\ge 30$. Beating the random-cut ablation means a statistically reliable AUC gain on departures that occur within $\Delta$ of a neighbor's departure.

Why not just…?

Why not just shuffle random windows with a long warm-up, ignoring cascades? This is the real threat. If the warm-up is long enough, it may already contain almost every chain, which would make the cut rule unnecessary. The random-cut ablation at matched warm-up length settles it. We also sweep the warm-up length to find whether cascade-closed cuts reach the same accuracy with a shorter prefix, because a shorter prefix directly lowers recomputation cost. If random cuts match at every prefix length, we will report that the gain comes from exchangeability alone and drop the cascade machinery.

Why not just keep DistTGL's contiguous time segments and compress the gradients with 4-bit or low-rank methods? SDP4Bit reports a 4.08x speedup on 128 H800 GPUs for an 18B model. SEPARATE reports roughly 2x faster GPT-2 Medium training. Both shrink dense gradients from large language models. Temporal graph networks are small, and DistTGL identifies node-memory synchronization as their dominant communication cost, which compression does not touch. In addition, Agarwal et al. find that the arithmetic of compression competes with backward-pass compute. Compression is orthogonal and could be stacked on our method. We include a 4-bit DistTGL variant to show the size of the residual bottleneck.

Why not just use static weekly snapshots with ordinary mini-batching? Snapshots are trivially interchangeable, but they discard the order within each week that the 5.05 same-week hazard ratio says matters. NeutronStream's gains over static batching point the same way. We include a snapshot graph network baseline under identical distribution.

Why not just rank users by degree to anticipate collapse? CKC shows that high-degree users are buffered by dense neighborhoods, while collapse is triggered at the core boundary. We report degree and PageRank rankings as non-learned references for the top-$b$ departure prediction task.

Proposed Experiments

Attrition tasks use the Coursera forum data of Yang et al. and the DistTGL MOOC and Reddit streams. They also use SNAP CollegeMsg, email-Enron and mathoverflow, which IncAVT already uses for dynamic core tracking. Departure labels are derived from sustained inactivity. DBLP serves as a check on chain statistics against CKC's reported linchpin cascade. GDELT tests scale and link prediction against DistTGL's published setting, and the NeutronStream suite covers the small-graph regime.

Metrics are departure AUC overall and on cascade-driven departures, link-prediction performance where DistTGL reports it, wall-clock time to a target AUC under throttled bandwidth, prefix recomputation as a fraction of total compute, and the weight-dispersion and averaging gap described above.

Three checks are decisive. The first sweeps $H$ over 1, 30 and 100 and should show contiguous-segment DiLoCo degrading as $H$ grows while shuffled cascade-closed windows stay within one AUC point of chronological single-machine training. If both degrade equally, exchangeability is not the mechanism. The second is the matched-prefix comparison of cascade-closed against random cuts, restricted to departures that occur mid-cascade. The third sweeps the warm-up length and tests whether accuracy saturates near the empirical chain-duration percentile. That would confirm that the cascade statistic, not a hand-tuned constant, sets the right window geometry.