Neural networks often learn in sudden jumps. A language model abruptly starts copying repeated patterns, or a masked language model suddenly starts tracking syntax, and the loss curve drops sharply. Researchers who study how networks change over training would like to know when such a jump begins, which inputs it affects, and whether it is visible inside the network before it shows up in the loss. Their current tools either compress each pair of checkpoints into one similarity number, which cannot say which inputs moved, or rely on persistent homology, a topology computation whose cubic cost limits it to a few hundred points and a handful of checkpoints.
We propose a cheaper and more informative instrument, taken from the first half of UMAP, a widely used method for drawing high-dimensional data in two dimensions. Before it draws anything, UMAP builds a weighted nearest-neighbor graph in which each point's distances are rescaled to its own local density. We build this graph on the activations of a fixed set of probe inputs at every saved checkpoint. We then measure how much the graph changes between consecutive checkpoints using UMAP's own fuzzy cross-entropy, a score that adds up edge by edge. The result is a "rewiring rate" curve over training. Because the score is a sum over edges, it also gives a per-input map of whose neighborhoods were rebuilt. The bet is that the main reorganizations of training appear as sharp peaks in this curve, that those peaks can come before the loss drop, and that the inputs responsible for each peak are the ones the new capability uses.
Core Claim
Let $\mu^{(t)}_{ij}$ be UMAP's symmetrized fuzzy membership strength between probe inputs $i$ and $j$, computed from layer-$\ell$ activations at checkpoint $t$. This is UMAP's $k$-nearest-neighbor graph with the per-point offset $\rho_i$ and scale $\sigma_i$ described in the UMAP tutorial and survey. We define the rewiring rate between checkpoints $t$ and $t' = t+\Delta$ as
$$R_\ell(t) = \frac{1}{|E|}\sum_{(i,j)\in E} \left[ \mu^{(t')}_{ij}\log\frac{\mu^{(t')}_{ij}}{\mu^{(t)}_{ij}+\epsilon} + (1-\mu^{(t')}_{ij})\log\frac{1-\mu^{(t')}_{ij}}{1-\mu^{(t)}_{ij}+\epsilon} \right],$$
where $E$ is the union of edges present in either graph and $\epsilon$ is a small constant for numerical stability. The per-input score $r_i(t)$ is the same sum restricted to edges that touch input $i$.
We make three claims. First, $R_\ell(t)$ peaks at known developmental events, and in some cases the peak comes before the loss drop. Second, the inputs with the highest $r_i$ at a peak are enriched for the inputs the emerging mechanism acts on. Third, removing the mechanism by causal intervention removes or moves the peak.
Two design choices separate this from simply drawing UMAP plots over time. The first is that every quantity is computed on the high-dimensional graph and never on the two-dimensional layout. This avoids the known artifacts of the layout step: the repulsion bias, the flattening of density differences, and the arbitrary rotations between runs (UMAP tutorial and survey; Aligned-UMAP longitudinal study). The second is that the per-point rescaling by $\sigma_i$ makes the score insensitive to global changes in scale and density. That matters because representations are known to expand and then compress during training (The Shape of Learning), and a raw-distance metric would mistake that cycle for reorganization.
Relation to Prior Work
The motivation comes from developmental interpretability. What needs to go right for an induction head? shows that the abrupt emergence of in-context copying comes from three subcircuits that each mature smoothly and then lock together at a threshold. Sudden Drops in the Loss ties a sharp loss drop in BERT to the onset of attention patterns that follow syntax, and shows that suppressing those patterns early changes the outcome. Evolution of Concepts in Language Model Pre-Training finds that learned feature directions rotate almost orthogonally around step 1,000. These papers locate events through scalar loss, task-specific probes, or expensive dictionary learning across snapshots. We ask whether a single task-agnostic neighborhood statistic recovers the same events and attaches them to specific inputs.
Several papers constrain the design. Representation Topology Divergence (RTD) compares point clouds that share sample correspondences using persistent homology. It tracks training epochs and beats the standard similarity metrics on synthetic cluster changes, but at a cost that limits it to roughly 500 to 1,000 points. It uses UMAP only as a preprocessing step. Similarity of Neural Network Representations Revisited establishes linear CKA, a kernel similarity score between two representations, as the standard. CKA returns one number with no per-input breakdown. Koopman-based methods (Identifying Equivalent Training Dynamics; Dynamical Similarity Analysis) fit linear operators to whole trajectories and compare their eigenvalue spectra. They capture global dynamics but say nothing about which data points reorganize.
On the UMAP side, Aligned-UMAP supplies a method we reuse, but only for visualization. That work aligns embeddings across clinical time slices; it does not quantify change, and it never touches network training. AP-MAE clusters attention patterns with UMAP, but only on finished checkpoints. Our contribution is to use UMAP's graph-construction step as a measuring device across training.
Research Program
We first build the instrument. For each checkpoint and layer we extract activations for a fixed probe set of 5,000 to 20,000 inputs and build UMAP's fuzzy graph with approximate nearest neighbors, which costs close to $O(N\log N)$. We then compute $R_\ell(t)$ and $r_i(t)$ across densely spaced checkpoints. We also sweep the neighborhood size $k$ to check that peaks are stable rather than tuned. Aligned-UMAP layouts are produced only as a qualitative companion, for displaying which inputs move at a peak.
We then test the instrument in four settings where the ground truth about developmental events is already established.
The first setting is the induction-head setup on Omniglot from What needs to go right for an induction head?. Here, clamping specific attention behavior during training gives causal control over when the circuit forms.
The second is BERT-Base trained on BookCorpus and English Wikipedia, following Sudden Drops in the Loss. Its syntax-suppression intervention lets us move the onset of syntactic attention.
The third is a ResNet trained on ImageNet-1k or CIFAR. The early spike and recession of class-selective neurons described in On the special role of class-selective neurons in early training provides an early, transient event.
The fourth is grokking on modular arithmetic, where the network memorizes its training set first and only generalizes much later. Identifying Equivalent Training Dynamics supplies runs that grok and runs that do not.
Baselines
The naive reference point is the training loss itself, smoothed and differentiated, with its largest negative curvature marked as the detected event. Any instrument that only re-finds the loss drop at the same step adds nothing for timing. Beating this baseline means detecting the event earlier, or detecting events that leave no mark on the loss, such as the subcircuit maturation before induction heads lock in and the class-selectivity spike.
The standard representation baseline is linear CKA between consecutive checkpoints, following Similarity of Neural Network Representations Revisited. We report it as $1-\mathrm{CKA}$ so it reads as a change curve, and compute it on exactly the same probe activations and checkpoints.
The strongest existing competitor is RTD from Representation Topology Divergence. It is also sample-matched and sensitive to nonlinear structure. On synthetic cluster tracking it reports a Kendall-tau correlation of 1.0, against 0.23 for CKA and 0.14 for SVCCA, and a correlation with model error under noise shift of 0.966 against 0.927 for CKA. For fairness, RTD runs on random subsamples of the largest size it handles, which that paper puts at roughly 500 to 1,000 points, repeated until its wall-clock budget matches ours. We also run our method on the same small subsample to separate the effect of sample size from the effect of the method. Beating RTD means matching its timing accuracy on the small subsample while scaling to many more checkpoints and points, and providing per-input localization, which RTD does not give. No reported result was found on Lacuna for RTD, or any other method, detecting the onset of induction heads, syntactic attention, or grokking.
Two ablations isolate the key ingredient. The first replaces the fuzzy graph with an unweighted $k$-nearest-neighbor graph scored by Jaccard turnover, which tests whether the density-adaptive weights matter at all. The second keeps the weights but drops the $\rho_i$ and $\sigma_i$ normalization, using raw Gaussian distances instead, which tests whether local rescaling is what shields the score from the expansion-and-compression cycle.
For the early-prediction question in the grokking setting, the comparison is the Koopman spectral signature from the first 100 iterations. That paper reports a qualitative separation but no classification score that we found on Lacuna.
Why not just…?
Why not just use CKA between checkpoints? CKA is cheap and invariant to rotations, but Representation Topology Divergence shows it reaches only 0.23 Kendall tau on synthetic cluster changes, so it largely misses the splits and merges we expect at transitions. It also cannot be broken down by input. If CKA nonetheless times the events as well as $R_\ell$ does, our timing claim fails, and only localization remains as a contribution. The head-to-head comparison in the first experiment settles this.
Why not just count how many nearest neighbors change? This is the most serious threat, because the plain neighbor-turnover score is simpler and could carry most of the signal. Our argument for the fuzzy version is that the local rescaling separates real rewiring from density changes driven by the expansion-and-compression cycle (The Shape of Learning). That argument is untested. The Jaccard ablation is therefore a primary result. If it performs equally well, we will report neighbor turnover as the recommended tool and frame UMAP's contribution as the specific choice of graph.
Why not just run UMAP on each checkpoint, or use Aligned-UMAP, and look at the pictures? Independent UMAP runs produce arbitrary rotations and layouts (Aligned-UMAP). Alignment fixes this but adds a regularizer that deliberately smooths out jumps, which is exactly the signal we want to measure. The layout step also distorts density (UMAP tutorial and survey). Measuring change on the high-dimensional graph avoids all three problems. We will also compute a version of $R_\ell$ on the two-dimensional layouts to show directly how much signal the layout step loses.
Why not just use Koopman methods? They summarize whole trajectories as eigenvalue spectra, which require choosing delay and rank settings and give no account of which inputs moved (Dynamical Similarity Analysis). We treat them as complementary. In the grokking setting we test whether early rewiring peaks predict grokking about as well as the Koopman signature does.
Proposed Experiments
Timing. For each setting we compare the step of the largest peak in $R_\ell$, together with all secondary peaks above a threshold set on shuffled-checkpoint nulls, against the ground-truth events. In the induction setting the events are the subcircuit metrics and the step at which in-context accuracy jumps. In BERT they are the onset of syntactic attention and the BLiMP trajectory, using BLiMP. In the ResNet they are the epoch of peak class selectivity. In grokking they are the jump in test accuracy. We report the timing error and how far each detection precedes the loss drop, against every baseline, across at least five seeds.
Localization. At each peak we rank inputs by $r_i$ and measure the area under the receiver operating characteristic curve (AUROC) for predicting membership in an independently defined target set. The target sets are tokens inside repeated sequences for induction heads, tokens with syntactic dependents for syntactic attention, and images of classes whose selective units fire most strongly for the class-selectivity spike. CKA and RTD cannot produce a per-input score, so the reference points here are random ranking and the Jaccard ablation.
Causal checks. The experiment that would decide the question is interventional. If the rewiring peak reflects the mechanism, then suppressing syntactic attention for the first 3,000 steps, as in Sudden Drops in the Loss, should move the BERT peak to after the suppression ends, and permanent suppression should remove it. Likewise, clamping the copying subcircuit in the induction setup should shift the peak by an amount we can predict. A peak that stays put under these interventions would show that the metric tracks something other than the mechanism, and would count against the proposal.
Scale. Finally, we report wall-clock time against the number of probe points for our method and for RTD. This checks whether running the instrument on every checkpoint is practical for a 100k-step BERT run.
References
- Uniform Manifold Approximation and Projection (UMAP) and its Variants: Tutorial and Survey
- Application of Aligned-UMAP to longitudinal biomedical studies
- The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models
- What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation
- Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
- Evolution of Concepts in Language Model Pre-Training
- Representation Topology Divergence: A Method for Comparing Neural Network Representations
- Similarity of Neural Network Representations Revisited
- Identifying Equivalent Training Dynamics
- Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysis
- Automated Attention Pattern Discovery at Scale in Large Language Models
- On the special role of class-selective neurons in early training
- BLiMP: The Benchmark of Linguistic Minimal Pairs for English