Abstract
Predictive coding (PC) networks are a biologically interesting class of neural networks. Their layered hierarchy mimics the reciprocal connectivity pattern observed in the mammalian cortex, and they can be trained using local learning rules that approximate backpropagation. However, despite having feedback connections that enable information to flow down the network hierarchy, discriminative PC networks are not generative. Clamping the output class and running the network to equilibrium yields an input sample that typically does not resemble the training input. This paper studies this phenomenon and proposes a simple solution that promotes the generation of input samples that resemble the training inputs. Simple decay, a technique already in wide use in neural networks, pushes the PC network toward a unique minimum 2-norm solution, and that unique solution provably (for linear networks) matches the training inputs. The method also vastly improves the samples generated for nonlinear networks, as we demonstrate on MNIST.
Introduction: The Generative Gap in Predictive Coding
In the quest for biologically plausible artificial intelligence, Predictive Coding (PC) stands out as a leading framework. Unlike standard deep learning models that rely on the computationally efficient but biologically questionable backpropagation algorithm, PC networks utilize local learning rules. Their hierarchical structure, characterized by state nodes and error-correcting neurons, closely mimics the reciprocal connectivity patterns found in the mammalian cortex.
Theoretically, PC networks possess the necessary architecture to be "generative." Because they feature feedback connections that carry signals down the hierarchy, one should be able to "clamp" the output layer (e.g., setting it to a specific category like "6") and allow the network to settle into an equilibrium that produces a corresponding image at the input layer. However, in practice, this process typically fails. Instead of recognizable images, standard PC networks in generative mode tend to produce meaningless noise or static.
The Ill-Posed Nature of Generation
The authors identify that the failure of PC networks to generate recognizable samples is not a structural flaw, but rather a result of the generative problem being ill-posed. In a typical classification task, the input space (e.g., thousands of pixels) is much larger than the output space (e.g., ten classes). When running the network backward, there are infinite possible input patterns that could satisfy a single output label. Without additional constraints, the network has no mathematical reason to choose a pattern that resembles the training data.
Methods: The Power of Simple Decay
To resolve this ambiguity, the authors propose a deceptively simple modification: Simple Decay. In neural network terms, this is equivalent to L2 regularization. By adding a decay term ($-\lambda x$ and $-\lambda M$) to the differential equations governing the update of state nodes ($x$) and weight matrices ($M$), the network is penalized for excessive activity or overly large weights.
The mathematical foundation of this approach is captured in Theorem 1. The authors prove that for a linear PC network, adding decay forces the system toward a unique "minimum 2-norm solution"—the solution that satisfies the output constraint while keeping the magnitude of the input as small as possible. Crucially, they demonstrate that this unique solution is exactly the training exemplar. By "shrinking" the space of possible solutions, the decay term forces the network to settle on the most representative representation it encountered during training.
Key Results
The effectiveness of simple decay was evaluated on both a linear toy dataset and the MNIST benchmark.
- Linear Accuracy: In the toy dataset, the correlation between the generated sample and the original training exemplar improved from 0.204 (without decay) to 0.995 (with decay).
- Nonlinear Success on MNIST: When tested on deep, nonlinear networks using
tanhactivation functions, the correlation improved from 0.630 to 0.979.
Visually, the difference is stark. While an unconstrained PC network produces a grey square of noise when prompted to generate a digit, the regularized network generates recognizable digits that closely resemble the training data.
Discussion and Biological Plausibility
While the addition of decay significantly improves generative performance, it comes with a "performance tax." The authors noted a 5% to 10% decrease in discriminative accuracy on MNIST, suggesting that the constraints required for high-quality generation may slightly limit the flexibility needed for optimal classification. Furthermore, simulating these networks is computationally intensive, as it requires solving systems of differential equations until they reach equilibrium.
Despite these trade-offs, the findings have significant implications for neuroscience. Weight and activity decay are common features in biological neural models. This research suggests that these biological "limitations" might not be mere metabolic constraints, but essential mechanisms that allow the brain to perform mental imagery and generate coherent internal representations.
Summary updated