Abstract
Predictive Coding Networks (PCNs) aim to learn a generative model of the world. Given observations, this generative model can then be inverted to infer the causes of those observations. However, when training PCNs, a noticeable pathology is often observed where inference accuracy peaks and then declines with further training. This cannot be explained by overfitting since both training and test accuracy decrease simultaneously. Here we provide a thorough investigation of this phenomenon and show that it is caused by an imbalance between the speeds at which the various layers of the PCN converge. We demonstrate that this can be prevented by regularising the weight matrices at each layer: by restricting the relative size of matrix singular values, we allow the weight matrix to change but restrict the overall impact which a layer can have on its neighbours. We also demonstrate that a similar effect can be achieved through a more biologically plausible and simple scheme of just capping the weights.
Introduction
Predictive Coding (PC) is a brain-inspired framework where neural networks learn to minimize the "prediction error" between layers. Unlike standard deep learning, which uses global backpropagation, PCNs use local updates, making them highly attractive for both neuroscience and efficient hardware implementations. However, researchers have identified a strange "pathology" in PCNs where classification accuracy peaks and then plummets with continued training. Crucially, this isn't standard overfitting—where the model memorizes the training data but fails on new data. In PCNs, accuracy drops on both training and test sets simultaneously. This paper diagnoses why this happens and how to stop it.
The Predictive Coding Architecture
The study utilizes a Generative Predictive Coding Network—a model that learns to generate images from labels. To classify an image, the network "inverts" itself, iteratively updating its internal state until it finds the label that best predicts the input image.
The study primarily uses the MNIST dataset, consisting of 70,000 handwritten digits. Training was conducted on 60,000 images in batches of 640, while testing used 10,000 images. While MNIST is a standard benchmark, it served as a clear environment to isolate and observe the specific mathematical pathology of the network.
Methods: Addressing the Accuracy Decay
To solve the accuracy decay, the authors tested two primary interventions:
- Weight Regularization: They modified the weight update rule to restrict the relative size of matrix singular values (or maintain a constant L1 norm), allowing the weight matrix to change while limiting the overall impact a layer has on its neighbors.
- Weight Capping: A simpler, more biologically plausible approach where individual weights are prevented from exceeding a specific threshold (e.g., 0.1).
They compared these against a standard PCN and an unsupervised version where the top layer isn't "clamped" to specific labels during training.
Results: Achieving Stability
By applying weight regularization or capping, the authors successfully prevented the accuracy decay. Without intervention, accuracy on MNIST peaked and then fell toward 60%. With regularization, it maintained its peak performance (approx. 85-90% for this specific generative architecture) indefinitely.
As shown in the results, the standard PCN accuracy dives after approximately 1,000 batches, while the regularized versions stay flat and high.
The Inference Speed Trap
The "accuracy drop" is identified as a "speed drop." As training progresses, the network still could reach the correct answer, but it takes more and more iterations to get there. Because researchers typically stop inference after a fixed number of steps (e.g., 1,000), the network appears to be getting dumber when it's actually just getting slower.
This is caused by a Supervised Mismatch: in supervised learning, the top layer is "clamped" to a specific label, which keeps the top-layer weights small. Meanwhile, the hidden layers are free to grow, creating a massive imbalance in gradient magnitudes. PCNs are Energy-Based Models (EBMs) where inference is the process of the network "settling" into a low-energy state. The authors found that the top layer takes increasingly long to settle because its influence is dwarfed by the growing weights of the layers below it.
Limitations and Future Work
- The Role of Precisions: The authors used identity matrices for precisions. Learning these precisions might naturally solve the imbalance, but this remains to be tested.
- Hyperparameter Sensitivity: The "weight cap" method is sensitive to the specific cap chosen (0.1 in this study).
- Scaling Up: It remains to be seen how these dynamics play out in much deeper networks or more complex datasets like CIFAR-10 or ImageNet.
Summary updated