When a neural network learns tasks one after another, it does not forget evenly. Its early layers, which detect edges and textures, stay almost unchanged. Its late layers, which hold task-specific logic, get overwritten. Separately, a long line of work on slow feature analysis (learning features that change slowly over time, on the theory that object identity persists while pixels flicker) shows that temporal continuity in video is a free, label-independent signal for learning stable, invariant features. This proposal joins the two observations. In continual learning from video, a slowness objective should be applied mainly at the layers where forgetting concentrates. It should also carry the decorrelation constraint that classic slow feature analysis uses to prevent collapse. The reason to expect it to work is concrete: a slowness objective compresses a layer's useful variance into a few persistent directions, and theory predicts that forgetting grows with representation dimensionality.
The test domain is the CORe50 continual object-recognition benchmark, which is recorded as continuous hand-held video sessions. Standard continual-learning pipelines shuffle these frames and throw the temporal structure away. We keep the session order, attach slowness losses to chosen depths of the backbone, and ask three questions. Does the benefit follow the measured depth profile of forgetting? Does it run through a reduction in the deep layers' effective dimensionality? And does it disappear when the temporal order is destroyed but everything else stays the same?
Core Claim
The claim is that temporal slowness reduces forgetting in a specific, testable way. Kim et al. (Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective) define representation discrepancy, the residual left after optimally linearly aligning a layer's features before and after later tasks. They show it grows with depth and saturates at a level proportional to the layer's representational dimensionality. On Split-CIFAR100 it correlates with linear-probe forgetting at $R^2 = 0.88$. A slowness loss with a whitening-style decorrelation constraint concentrates variance into a small number of slow, object-tied directions. We therefore predict that a slowness term placed at depth $k$ will lower the effective dimensionality of layer $k$, and that this will lower that layer's discrepancy and forgetting roughly in proportion. The effect should be largest at deep layers, smallest at shallow ones, and absent when frame pairs are drawn without temporal adjacency.
The prediction can fail in two ways, and each failure would teach us something. First, a plain rank-limiting regularizer with no temporal pairing might give the same gain. In that case slowness is just dimensionality reduction, and the temporal ingredient is unnecessary. Second, slowness at deep layers might cut forgetting while also cutting accuracy on new tasks. That would resolve the open tension between invariance and plasticity in the pessimistic direction: forcing top layers to be invariant suppresses their ability to absorb new task boundaries.
Relation to Prior Work
Two bodies of work motivate the depth-targeting. The first establishes that forgetting is concentrated in deep layers. Anatomy of Catastrophic Forgetting tracks layer-wise similarity across tasks and shows lower layers stay near-identical while upper layers drift; it also shows that freezing lower layers costs little. Auxiliary Classifiers shows that intermediate layers still classify old tasks after the final layer has forgotten them. Kim et al. supply the dimensionality account we turn into a prediction. The second establishes that this depth profile depends on architecture: GPT-2 shows a U-shaped profile (Nandan and Blondin). We therefore measure the profile for each backbone rather than assume "deeper means worse." Two current online continual-learning methods exploit the depth gradient with supervised objectives. MOSE attaches classifiers at several depths and distills from stable shallow blocks into volatile deep ones. MuFAN aggregates multi-depth features and preserves angular structure between samples across tasks. Both treat frames as unrelated static images. We add an unsupervised temporal signal and use it to choose depths.
The slowness literature supplies the method. Classic slow feature analysis (Bray et al.) defines the objective with unit-variance and decorrelation constraints. Incremental Slow Feature Analysis makes it online and hierarchical. DL-SFA stacks two learned layers of it. Latent Predictive Learning pairs a slowness "pull" with a variance "push" in every layer of a deep network. Temporal Slowness in Central Vision uses nearby video frames as contrastive positives on Ego4D, improving instance recognition from 59.03% to 67.00%. All of these apply slowness uniformly or only at the output. None of them tests sequential tasks or measures forgetting.
The closest work combining the two directions is DualNet. It trains a "slow" backbone with static Barlow Twins on a replay buffer. Its "slow" refers to a slowly updated learner in the complementary-learning-systems sense, not to slowly varying features. It uses image augmentations rather than temporal neighbors and treats the backbone as one block. Our proposal differs in the training signal (temporal adjacency), in where the signal is applied (chosen depths), and in the mechanism we test (dimensionality mediating forgetting).
Research Program
The model is a reduced ResNet-18 trained online, in a single pass, on CORe50 in its New Classes (NC) and New Instances and Classes (NIC) protocols. We keep within-session frame order, so each incoming minibatch contains short runs of temporally adjacent frames. All arms share a small reservoir replay buffer, so the slowness term is tested on top of replay rather than in place of it. At each candidate depth $k$ (the outputs of the four residual stages) we attach a small projection head $g_k$ and add
$$\mathcal{L}_k = \mathbb{E}_t \left\| g_k(x_t) - g_k(x_{t+\tau}) \right\|^2 + \beta \left\| C_k - I \right\|_F^2.$$
Here $x_t$ and $x_{t+\tau}$ are frames $\tau$ steps apart in the same session. $C_k$ is the batch covariance of the standardized head outputs. $I$ is the identity matrix. $\beta$ weights the decorrelation penalty. The first term is the finite-difference version of the slow feature analysis cost $\mathbb{E}[(\partial y/\partial t)^2]$. The second term replaces its unit-variance and decorrelation constraints with the cross-correlation-style penalty used in deep self-supervised learning. The total loss is cross-entropy plus replay plus $\sum_k \lambda_k \mathcal{L}_k$.
The study has three parts. The first is a placement sweep: slowness at a single depth (each of the four stages), uniformly at all depths, and graded with $\lambda_k$ increasing with depth. The second is an adaptive variant. Every few hundred steps it measures each stage's drift as linear CKA between current features and features stored at the last task boundary, both computed on replay samples. It then sets $\lambda_k$ proportional to $1 - \mathrm{CKA}_k$, so the backbone itself reports where slowness is needed. If the measured profile is not monotone, this variant handles it. The third is a mechanism analysis. For every run we log, per layer and per task boundary, the effective dimensionality (participation ratio of the feature covariance eigenvalues), the representation discrepancy, and linear-probe accuracy on old tasks. We then test whether the slowness-induced drop in effective dimensionality statistically mediates the drop in discrepancy, as the linear-discrepancy theory predicts.
Baselines
The naive reference is online fine-tuning with the same replay buffer and no auxiliary loss. CORe50's own paper reports that plain SGD in the New Classes scenario falls to near zero without protection and that the CWR baseline holds 35–40% accuracy, against a cumulative upper bound of about 75% (CORe50). We rerun replay-only fine-tuning and a jointly trained upper bound under our exact protocol so every arm is compared on identical splits.
The strongest existing competitors are MOSE and MuFAN. MOSE is the most direct rival because it already exploits the shallow-stable, deep-volatile gradient. It reports 55.6% average accuracy on online Split CIFAR-100 and 39.3% on Split Tiny-ImageNet. MuFAN is the strongest method with a CORe50 result in the file: a 17% relative improvement over prior methods, with no absolute accuracy given. We found no reported result on Lacuna for MOSE or DualNet on CORe50 in absolute terms, so both will be rerun with their released hyperparameters. Each gets the same backbone, buffer size, and single pass, plus a tuning budget matched to ours (the same number of hyperparameter trials on a held-out validation session). Beating them means higher final average accuracy and lower forgetting, with non-overlapping 95% intervals over five seeds, at equal or lower training compute. We also test whether adding our deep-layer slowness term to MOSE improves MOSE. That is the more informative outcome, because it shows the temporal signal is complementary to multi-depth supervision.
The DualNet-style competitor is static self-supervision on the replay buffer: Barlow Twins on two augmentations of the same frame, at the same depths and weights as our slowness term. This isolates whether temporal pairs beat augmentation pairs.
Two ablations isolate the key ingredient. The shuffled-pair control uses exactly our loss, but $x_{t+\tau}$ is a random frame of the same object from another session. This keeps the supervision-free label structure and the decorrelation but removes temporal adjacency. The rank-matched control drops the slowness term and instead caps layer $k$'s effective dimensionality, using a nuclear-norm penalty tuned to match the effective dimensionality our method reaches. Beating the shuffled control shows temporal order matters. Beating the rank-matched control shows slowness does more than shrink dimensions.
Why not just…?
Why not just use static self-supervised learning on replayed images, as DualNet does? DualNet's 73.20% accuracy and 3.86% forgetting on task-aware Split miniImageNet show static self-supervision helps, so this alternative is a real threat. Our argument for temporal pairs rests on the Ego4D evidence that adjacent frames supply real viewpoint changes that augmentations cannot, but that evidence comes from offline pretraining, not continual learning. The augmentation-pair arm settles the question directly. If it matches temporal slowness on CORe50, we will report that temporal order adds nothing for forgetting, and the contribution reduces to the depth-placement result.
Why not just freeze the early layers? Freezing protects exactly the layers that were not forgetting. Anatomy of Catastrophic Forgetting shows freezing the lower half preserves learning ability, but drift in the unfrozen deep layers remains. Freezing is therefore complementary to our method, not a substitute. We include a frozen-early-stages variant of both replay and our method to check that the gains survive when shallow layers are already fixed.
Why not just distill or supervise at multiple depths, as MOSE does? MOSE already gets large online gains, and the multi-depth classifiers studied in Auxiliary Classifiers improve ten baselines by about 10% relative. Supervised signals, however, come only from labeled current data and a small buffer, which MOSE itself identifies as prone to buffer overfitting. Slowness uses every incoming frame pair without labels. If MOSE alone matches MOSE-plus-slowness, the unsupervised term is redundant and we will say so.
Why not just reduce dimensionality directly? If our mechanism is right, a rank penalty should recover part of the benefit. The rank-matched control measures how much. A full match would mean the theory is right but the temporal signal is incidental. Most of the gain surviving only with temporal pairs would mean slowness picks out the right low-dimensional subspace, the object-tied one, rather than any small subspace.
Decisive Checks
The main experiment is CORe50 NC and NIC, online, with five seeds. We report final average accuracy, average forgetting, and accuracy on each newly arrived task (to measure plasticity). The placement prediction holds if gains rise with depth in proportion to each stage's measured drift, and if the adaptive variant matches or beats the best fixed placement. The mechanism prediction holds if effective dimensionality and discrepancy at the treated layer fall together, and if the dimensionality change accounts for most of the forgetting reduction across runs. The shuffled-pair control must lose most of the gain.
Two further checks guard against flattering ourselves. The first is a negative control on Split CIFAR-100, which has no temporal structure. There our temporal pairs reduce to augmentation pairs, so the method should give no advantage over the static self-supervision arm. A clear win there would mean something other than temporal slowness is driving the CORe50 result. The second is a background test. Temporal slowness within a session may bind the session's background to the object, the object-versus-scene trade-off noted in the central-vision work. We therefore measure accuracy on objects seen against held-out session backgrounds. Deep-layer slowness that helps within sessions but hurts across them would be a reportable failure mode. Its remedy, a shorter $\tau$ or cropping to the hand-held object, would be tested as a follow-up.
References
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective
- Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
- Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
- Universal Neurons in GPT-2: Emergence, Persistence, and Functional Impact
- Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation
- New Insights for the Stability-Plasticity Dilemma in Online Continual Learning
- Kernel-Based Extraction of Slow Features: Complex Cells Learn Disparity and Translation Invariance from Natural Images
- Incremental Slow Feature Analysis
- DL-SFA: Deeply-Learned Slow Feature Analysis for Action Recognition
- The combination of Hebbian and predictive plasticity learns invariant object representations in deep sensory networks
- Temporal Slowness in Central Vision Drives Semantic Object Learning
- DualNet: Continual Learning, Fast and Slow
- CORe50: a New Dataset and Benchmark for Continuous Object Recognition