Unlearning Without a Backward Pass in Predictive Coding Networks

Most machine unlearning methods, which remove the effect of specific training examples from a trained model without retraining it, depend on backpropagation. They use it to find which weights hold the unwanted memory, to compute the directions that protect everything else, and to take the unlearning steps themselves. Brains and much low-power neuromorphic hardware have no backward pass. They learn from local signals only: the activity on either side of a synapse and a local error. That raises a question the literature has not asked. Can a network that learns only from local signals also forget selectively, deleting one class or one subset of examples without damaging the rest of what it knows?

We propose to study this in predictive coding networks. In these networks, each layer tries to predict the activity of the layer below. Learning works in two phases: the network first lets its activities settle to reduce total prediction error, then applies local, Hebbian-style weight updates. Our central bet is that the settling phase is itself a tool for unlearning. Suppose the output is held firmly at a "replacement" label for a forgotten example. The network then redistributes the resulting error through every hidden layer before any weight changes. That tells us, using only local quantities, which neurons carry the memory, and it rewrites hidden representations rather than only the classifier. If this works, forgetting in such networks should reach inside the network instead of being cosmetic at the output.

Core Claim

We claim that a predictive coding network can unlearn using only local signals, and can do it about as well as backpropagation-based unlearning run on the same network. The procedure has three parts. We strongly clamp the output of forget examples to their next-most-likely class, borrowing the pseudo-labeling idea from Machine Unlearning via Null Space Calibration. We let inference settle. We then apply local weight updates only to the neurons whose settled prediction errors are specific to the forget set. Our second claim is sharper and falsifiable: this local procedure leaves less recoverable information in intermediate layers than output-focused methods. We measure this with linear probes, following An Illusion of Unlearning?, which showed that probes on internal features of backpropagation-trained networks recover roughly 70 to 80% forget accuracy even when output forget accuracy is 0%.

The mechanism we test has a specific source. Song et al. showed that under weak output nudging, predictive coding's local errors reproduce the backpropagation gradient. Weak-nudge predictive coding unlearning should therefore behave like gradient-based unlearning. The departure should come from strong clamping. In that regime Understanding Predictive Coding as a Second-Order Trust-Region Method showed that inference solves a trust-region problem scaled by the Fisher information, a measure of how sensitive the model's predictions are to each parameter, so the error is spread across layers in a curvature-aware way. The contrast between strong and weak clamping is therefore the key ablation of the whole proposal.

Over-parameterization enters as the regime that makes this hard and interesting. Machine Unlearning under Overparameterization argues that networks which fit their training data exactly have near-zero loss gradients on memorized points. Such networks also have many weight configurations that give identical outputs on the data we keep. Both facts let naive methods satisfy output metrics while traces survive in parts of the network the metrics never examine. We test whether local, layerwise errors avoid this failure. We do not assume they do. Clamping to a pseudo-label also gives backpropagation a non-vanishing gradient, so any advantage has to come from where the error lands, not merely from its being nonzero.

Research Program

The substrate is predictive coding networks trained with the parameterization of μPC, because unscaled predictive coding fails beyond about 15 layers. We use multilayer perceptrons on MNIST and Fashion-MNIST and ResNets on CIFAR-10 and CIFAR-100. μPC (the same work, indexed under a second id) reports training a 128-layer ResNet locally and reaching about 98% on MNIST, but no reported result on Lacuna for μPC's final CIFAR-10 accuracy was found. Establishing that base accuracy is therefore our first step. All comparisons are made within the same trained network, so a weaker base model does not confound the unlearning comparison.

The local localization score mirrors the importance difference used in Unlearning with Fisher Masking, but builds it from settled local quantities rather than backpropagated gradients. The procedure has four steps.

  1. For each hidden neuron $j$, we run inference with the output strongly clamped to a pseudo-label.
  2. We record the local Hebbian update magnitude $u_j(x) = \lVert \varepsilon_j(x)\, a_j(x)^\top \rVert$. Here $\varepsilon_j$ is the neuron's settled prediction error, and $a_j$ is the settled presynaptic activity feeding it.
  3. We compute the score $$ s_j = \mathbb{E}_{x \in D_f}[u_j(x)] - \lambda\, \mathbb{E}_{x \in D_r}[u_j(x)], $$ where $D_f$ is the forget set and $D_r$ is a retain sample, also clamped to pseudo-labels. The retain term measures how much neuron $j$ would have to change to flip examples we want to keep. The weight $\lambda \geq 0$ trades forgetting against protection.

  4. We apply the unlearning update only to the top-scoring fraction $\rho$ of neurons. Each forget-example update is interleaved with retain examples clamped to their true labels, which acts as a local analogue of the retain term in RMU from The WMDP Benchmark.

No global gradient, Hessian, or singular value decomposition is computed at any point. Every quantity is available at the synapse after settling.

A secondary program asks whether the procedure helps ordinary networks. Predictive coding inference can be run on any feedforward network by treating its layers as predictors. We therefore also apply the procedure to backpropagation-trained ResNets and treat it as an unlearning optimizer. If it beats NegGrad+ and UNSC on probe-level forgetting there, the result matters beyond biological plausibility.

Baselines

The naive reference points are doing nothing, measured as forget, retain, and test accuracy and probe accuracy of the original model, and plain local fine-tuning on the retain set with the same predictive coding learning rule. Unlearning with Fisher Masking found that ordinary fine-tuning left over 90% forget-class accuracy for GoogLeNet on CIFAR-100. We expect the local version to fail similarly, and confirming this validates the setup.

The gold standard is retraining from scratch on the retain set with μPC. It defines target forget accuracy, membership-inference behavior, and probe accuracy. FisherMask reports retraining averaging 91.7 epochs against 2.1 epochs of fine-tuning after masking. We report compute as wall-clock time and settling iterations, because the $K$ inference steps per example are predictive coding's real cost.

The strongest existing competitors are UNSC (Machine Unlearning via Null Space Calibration) and DEL (Improved Localized Machine Unlearning Through the Lens of Memorization). UNSC projects updates into the null space of retain-set activations and pseudo-labels forget samples. It reports raising CIFAR-10 retain accuracy from 92.10% to 92.77% while driving forget accuracy to 0.70%. DEL resets neurons scored by $|\theta_j g_j|$ and fine-tunes them on retain data. We use the official DEL-Unlearning codebase, which also implements NegGrad+ and SalUn. We also run MinNorm-OG, FisherMask, and NegGrad+ with class-mean-feature alignment (CMF, which forces classifier weights to match class feature means) from An Illusion of Unlearning?. That paper reports NegGrad+ with CMF lowering feature-level forget accuracy from 57.43% to 41.54% on CIFAR-100 with 10 classes forgotten.

All of these backpropagation methods are run on the very same μPC-trained network, using autograd through its feedforward pass. They get the same forget and retain data, the same retain-sample budget, and the same number of hyperparameter trials, tuned on a held-out validation forget split. Beating them has two parts. First, output forget and retain accuracy must fall within one point of the best of UNSC and DEL. Second, linear-probe forget accuracy must fall strictly below theirs and move toward the retrained model's value.

Two ablations isolate the key ingredients. The first is the weak-clamp version of the same procedure, which is approximately backpropagation by Song et al.. The second replaces the localization mask with random neurons of equal count.

Why not just…?

Why not just run UNSC or DEL on the predictive coding network with ordinary autograd? This is the most serious alternative, and it is exactly our upper-bound baseline. If it wins on every metric, including probe-level forgetting, then local unlearning is only a hardware-compatibility result. That result is still useful for chips without a backward pathway, but it is not a better unlearning method, and we will report it as such. The experiment decides this directly.

Why not just use weak nudging, which already equals backpropagation locally? Then any benefit would be backpropagation's benefit, and the Fisher trust-region reading of Innocenti et al. would contribute nothing. We run weak and strong clamping with matched step budgets. If the probe-level gap between them is under two points, we drop the mechanistic claim.

Why not just partition the network at training time, as the lifelong predictive coding network does, so deletion means switching off a subnetwork? Lifelong Neural Predictive Coding routes tasks into sparse subnetworks by lateral inhibition, meaning neurons within a layer compete to be active. It reports about 98% on Split MNIST with near-zero backward transfer, where backward transfer measures how much learning new tasks changes accuracy on old ones. For deleting a whole class or task that was planned in advance, this is a real threat and may be cheaper. We include it on Split MNIST and Split Fashion-MNIST. Its weakness is forgetting arbitrary example subsets, or classes whose features are shared across routes. We test that with random-subset forgetting and with non-IID subsets drawn from within a class, following DEL's CIFAR-10 protocol.

Why not just prune the neurons the score selects, with no update? The over-parameterization literature suggests heavy pruning is often harmless. We include "mask only" as an ablation. If it matches the full method, then the localization score, not the local update, is the contribution.

Proposed Experiments

The first experiment is class removal and random 10% subset forgetting on CIFAR-10 and CIFAR-100 with μPC ResNets, plus Fashion-MNIST multilayer perceptrons as a sanity check. We report five measures:

  • forget, retain, and test accuracy;
  • a membership-inference score (how well an attacker can tell whether an example was in the training data);
  • linear-probe forget accuracy at every block;
  • alignment between class-mean features and the classifier;
  • a relearning test: forget accuracy after a few epochs of fine-tuning on a handful of forget examples, since recovery after brief fine-tuning is a known failure of concept erasure.

The second experiment scales depth and width at fixed data to test the over-parameterization hypothesis. The question is whether the probe-level advantage over UNSC and DEL grows as the network becomes more redundant.

The third experiment transfers the procedure to backpropagation-trained ResNet-18 on CIFAR-10, compared against the same baselines.

The decisive checks are three. First, the strong-clamp local method must match the best backpropagation baseline on output metrics within one point. Second, it must beat that baseline by at least five points on block-level probe forget accuracy. Third, it must beat its own weak-clamp and random-mask ablations. Failing the first check means local unlearning costs accuracy. Failing the second or third means predictive coding dynamics add nothing beyond locality, and we would report that as a negative result on the mechanism.