Methods that protect important weights against forgetting, such as Elastic Weight Consolidation and Synaptic Intelligence, share a hidden dependency. They need someone to announce "the task is over" so they can take a snapshot of the weights and fix which ones matter. Real data streams make no such announcements. Classes fade in and out, drift, and recur. This note proposes to take the announcement away from the outside world and hand it to individual units of the network. The mechanism borrows from a biological model called synaptic tagging and capture. Each weight keeps a short-lived, decaying "tag" that records how much it has recently helped reduce the loss. When the tags on one unit's incoming weights add up past a threshold, that unit alone "captures" them. It locks in their importance and moves its reference point, the snapshot of weights that the network is later penalized for drifting away from. Brief or noisy episodes decay before they ever trigger a capture. Sustained learning does trigger one, and different units consolidate at different times.
The main claim is testable: on streams with no task boundaries and no stored examples, this event-driven, unit-local consolidation forgets less than two alternatives. The first is the obvious boundary-free version of Synaptic Intelligence, which anchors weights to a smoothly moving average. The second is a single network-wide boundary detector. The claim also predicts robustness to transient bursts of off-distribution data that smooth anchoring absorbs. If a continuously updated moving-average anchor matches it, the claim is false. The experiments are built to show that cleanly.
What exists and where the gap is
Synaptic Intelligence (Intelligent synapses for multi-task and transfer learning) supplies the method we reuse. It computes each weight's contribution to loss reduction online, as a running sum of gradient times weight change. However, it consolidates that sum and snapshots the reference weights only at externally given task boundaries. Without boundaries, the reference point is undefined.
Two lines of work remove the boundary in different ways. Task-Free Continual Learning detects plateaus in a sliding-window loss and triggers a global importance update. Its authors concede that the window and threshold need tuning. Gradual streams such as the Gaussian streams of Online Boundary-Free Continual Learning by Scheduled Data Prior may never produce a clean plateau. SYNERgy sidesteps snapshots entirely. It anchors weights to an exponential moving average (EMA) of the network and estimates importance stochastically online. It is the closest existing work. It differs from this proposal in three ways: its consolidation is synchronous and global, its anchor moves continuously at a fixed rate, and it depends on a 200-example replay buffer.
CLS-ER and SDP also consolidate globally, through model-level averaging or distillation, and also rely on replay. SDP's own finding motivates us. A plain EMA puts its heaviest weight on the most recent past, so it absorbs fresh noise. A prior that peaks at an older, settled state works better. Tag-and-capture reaches a similar effect per unit rather than per model: the anchor stays put until evidence accumulates.
The biological models motivate the mechanism but do not supply a deep-learning version of it. Tag-Trigger-Consolidation and the TagTriC model show how decaying local tags plus a thresholded cell-wide trigger filter out transient stimulation. They are single-neuron simulations whose trigger is a whole-cell scalar. Bio-inspired continual learners that do scale to benchmarks are constrained differently. NNA with the inelastic rule is rehearsal-free but uses task-incremental protocols and a shallow network of about 100k parameters. Artsy needs task indices to create its gates. The review Theories of synaptic memory consolidation flags a constraint we must respect: consolidation that only accumulates eventually exhausts plasticity.
Core mechanism
Let $\theta_k$ be weight $k$, $g_k$ its loss gradient, and $\Delta\theta_k$ its update at a step. Each weight carries a tag $h_k$ that accumulates Synaptic Intelligence's work term but decays with time constant $\tau$:
$$h_k \leftarrow (1 - 1/\tau)\, h_k - g_k \Delta\theta_k .$$
Weights are grouped by the unit they feed, meaning one output channel or neuron, indexed $j$. A unit fires a capture event when its total positive tag $P_j = \sum_{k \in j} \max(h_k, 0)$ exceeds a threshold. When it fires, and for that unit only, each weight's importance is updated as $\Omega_k \leftarrow \gamma\,\Omega_k + \max(h_k,0)/(\delta_k^2 + \xi)$. Here $\delta_k$ is the weight's displacement since its last capture, $\xi$ is a small constant, and $\gamma \le 1$ is a slow decay. The unit's reference weights are then reset to the current values $\tilde\theta_k \leftarrow \theta_k$, and its tags are zeroed. Training minimizes the task loss plus $c\sum_k \Omega_k(\theta_k-\tilde\theta_k)^2$, as in Synaptic Intelligence.
Two design choices address known failure modes. First, the threshold is scale-free: a unit fires when $P_j$ exceeds a multiple $\kappa$ of the running median of $P$ across units in its layer. This avoids the absolute thresholds that make plateau detectors brittle. Second, the decay $\gamma$ lets importance that is never re-captured fade slowly. This is our answer to plasticity exhaustion, and we test it explicitly rather than assume it works.
The memory cost matches Synaptic Intelligence, which already stores two auxiliary numbers per weight (running work and consolidated importance). The added cost is one extra scalar per weight and one per unit.
Research Program
The work has three parts. First, we implement the rule in standard ResNet-18 learners trained online with backpropagation. The primary regime is rehearsal-free, because that is where the gap is and where no reported task-free baseline exists in our material. Second, we plug the rule into SYNERgy in place of its online Fisher estimate and EMA anchor, with a buffer of 200. This asks whether event-driven anchoring helps even when replay is present. Third, we analyze when and where capture events happen: whether they cluster near true distribution shifts, whether early and late layers consolidate on different schedules, and how they respond to injected transient noise.
Baselines
The naive reference is plain online fine-tuning with no protection. It sets the floor, and every method must clear it by a wide margin in the rehearsal-free regime.
The most important competitor is continuous Synaptic Intelligence with an EMA anchor. It keeps the decaying path-integral importance but updates importance and a per-weight EMA reference every step, with no trigger. It is the obvious boundary-free fix, essentially SYNERgy's anchoring without replay. It is also the ablation that isolates our key ingredient, the thresholded capture event. We tune its EMA rate and $c$ over the same grid size and compute budget as our $\kappa$, $\tau$, and $c$. Tag-and-capture beats it only if final accuracy is higher and forgetting lower on at least the Gaussian and periodic streams across seeds, beyond seed variance.
A second ablation keeps our tags but uses a single network-wide trigger, which fires when the summed tag across all units crosses the threshold. It isolates the contribution of per-unit asynchrony from the contribution of event-based anchoring.
Among existing task-free consolidation methods, the strongest is task-free Memory Aware Synapses with loss-plateau triggering (Task-Free Continual Learning). We run it with its 100-sample hard buffer and also without it, to keep the rehearsal-free comparison fair. An oracle version of Synaptic Intelligence given true boundaries, on streams that have them, serves as a reference ceiling rather than a competitor.
In the replay regime, the strongest methods are SYNERgy and SDP. The file reports SYNERgy at 30.00% accuracy on GCIL-CIFAR-100 with buffer 200 (SYNERgy), against CLS-ER at 25.06% and DER++ at 18.84%. SDP is reported at 76.29% final accuracy on the CIFAR-10 Gaussian stream, against DER++ at 60.54% (SDP). We rerun both in our codebase with the same buffer, backbone, and tuning budget before comparing. Beating them means matching or exceeding these reproduced numbers when our rule replaces SYNERgy's consolidation term.
For the rehearsal-free regime on these streams, we found no reported result on Lacuna for any rehearsal-free method on GCIL-CIFAR-100 or the Gaussian streams. The comparison there rests entirely on our reruns.
Why not just…?
Why not just anchor Synaptic Intelligence to a moving average and skip the trigger? This is the real threat, and it may work nearly as well. SDP's evidence gives a reason to doubt it: a single-rate average over-weights the immediate past and absorbs recent noise (SDP). With a fixed EMA rate, protection of old knowledge also erodes at a constant rate whether or not new learning is happening. Neither argument is proof. The matched ablation above settles the question. The noise-burst experiment below is designed to separate the two methods where their predictions differ most. If the ablation ties, we would report the trigger as unnecessary.
Why not just detect boundaries globally and run ordinary consolidation? Plateau and confidence detectors force discrete segments onto continuous change. They need tuned windows and thresholds (Task-Free Continual Learning), and confidence signals are unreliable in poorly calibrated networks (Online Continual Learning for Interactive Instruction Following Agents). The network-wide-trigger ablation and the plateau baseline measure directly whether per-unit timing matters on gradual and periodic streams, where a single clock should mis-time consolidation for some layers.
Why not just keep a replay buffer? Replay works, and in the replay regime we compete with it rather than replace it. But the systems in our material that handle boundary-free streams well (SYNERgy, CLS-ER, SDP, CoPE) all store raw examples. Storage is often forbidden by privacy or memory limits (NNA). Our primary claim concerns the setting where replay is not allowed, and the second part of the program reports honestly whether the rule adds anything once replay is present.
Proposed Experiments
Streams include CIFAR-10 and CIFAR-100 Gaussian and Periodic-Gaussian streams following SDP, GCIL-CIFAR-100 in both uniform and long-tail variants, MNIST-360, and Rotated MNIST. Split-MNIST and Split-CIFAR-100 serve as streams with boundaries, where the oracle reference applies. Metrics are final and anytime accuracy, plus SDP's Knowledge Loss Ratio and Knowledge Gain Ratio, which do not require task indices. We use five seeds.
The first decisive check injects transient bursts into the Gaussian stream: short windows of label-shuffled or off-distribution images. The prediction is that tag-and-capture fires few or no captures during bursts and loses little accuracy afterward. The EMA-anchored variant, in contrast, should drift its anchor toward the corrupted weights. The second check is the capture-timing analysis. If captures bear no relation to true shifts and no layer-wise structure appears, the biological story is decorative even if accuracy improves. The third check runs a stream ten times longer than standard with recurring classes. It tracks how fast the network learns new classes over time, with and without importance decay $\gamma$, to test whether consolidation exhausts plasticity. The fourth check sweeps the threshold multiple $\kappa$ across an order of magnitude and across stream speeds, to see whether the scale-free threshold escapes the brittleness that affects plateau detection.
References
- Intelligent synapses for multi-task and transfer learning
- Task-Free Continual Learning.
- Online Boundary-Free Continual Learning by Scheduled Data Prior
- SYNERgy between SYNaptic Consolidation and Experience Replay for General Continual Learning
- Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System
- Tag-Trigger-Consolidation: A Model of Early and Late Long-Term-Potentiation and Depression
- Modeling plasticity across different time scales: the TagTriC model
- Improving Performance in Continual Learning Tasks using Bio-Inspired Architectures
- Brain-inspired continual pre-trained learner via silent synaptic consolidation
- Theories of synaptic memory consolidation and intelligent plasticity for continual learning
- Online Continual Learning for Interactive Instruction Following Agents