Automated architecture search scores candidate networks cheaply before training them. The common scoring rules assume the network will be trained by backpropagation with signed errors. Brain-inspired networks break that assumption. In these networks every signal, errors included, must be a non-negative firing rate, and learning happens through slow iterative settling. Training one such network costs many times more than training an ordinary one, so exhaustive search is out of reach. The cheap scores may also point to the wrong architectures, because what makes a topology good under backpropagation need not make it good under a rule whose error signals can only be positive ratios.
This note proposes to test that directly and to build a better score. We will train a few hundred small cell architectures from a standard search benchmark with a biologically constrained predictive coding rule. Predictive coding here means a network that learns by comparing top-down predictions with bottom-up activity. We will then ask two questions. First, how well do backpropagation-based rankings and scores transfer to this rule? Second, does a score computed from one settling pass of the constrained rule itself, at initialization, predict final accuracy better? That score is a "learning-rule kernel" that measures how distinguishable the rule's per-example weight updates are. The contribution is either a working zero-cost score for this class of models, or a clear, measured finding that backpropagation rankings already suffice.
Core Claim
The claim is that architecture quality under a non-negative learning rule depends on two properties that backpropagation-based scores do not see. The first is whether the topology keeps firing rates away from zero. The second is whether ratio-style error signals survive the trip to early layers. A score computed from the rule's own update signals should capture both, and should rank architectures better than any backpropagation-derived score at equal compute.
The mechanism comes from Tightening the Biological Constraints on Gradient-Based Predictive Coding. That paper shows that non-negative predictive coding needs two ingredients: a positive bias on predictions, so that silenced units can be revived, and "division mismatch" errors (activity divided by prediction, modeling shunting inhibition) in place of subtraction. With these, plus Kolen-Pollack or random feedback weights, it matches backpropagation on MNIST, with 1.9% and 2.2% error against 1.9% for backpropagation. That paper uses fixed architectures. We use its rule as the training procedure inside a search.
Relation to Prior Work
Several papers motivate the idea by showing that non-negativity changes what a good network looks like. The only prior bridge between these areas is Neural Architecture Search for Spiking Neural Networks. It searches spiking architectures without training and finds that the input-separability score of Neural Architecture Search without Training is biased by the abundance of zeros in sparse spike codes. It fixes this with a sparsity-aware Hamming distance. That work, however, still trains the found networks with backpropagation-style surrogate gradients, and it scores only forward activity. Our proposal targets continuous non-negative rates trained by a non-backpropagation rule, and scores the learning signal rather than the activity. Spiking Network Initialisation and Firing Rate Collapse shows that activity in deep non-negative networks can fall to near 0 Hz by the 10th layer, and that gradient variance can shrink to $10^{-40}$. Gradient-norm scores computed on such networks would therefore measure numerical noise, and a collapse check has to be part of any score.
Other work supplies the method. The zero-cost scores of Zero-Cost Proxies for Lightweight NAS are pruning saliency metrics (Synflow, SNIP, Fisher, grad_norm) and Jacobian covariance, computed from one minibatch at initialization. They reach a Spearman rank correlation of 0.82 on NAS-Bench-201 when combined. We reuse their protocol and their scores as baselines. The log-determinant kernel of Neural Architecture Search without Training supplies the functional form of our score; we apply it to per-example weight updates instead of binary activation codes. The fixed-budget idea of Rethinking Performance Estimation in Neural Architecture Search cuts training epochs, layers and resolution while preserving rank, and reports roughly 1,000× savings. It supplies both our reduced-fidelity ground-truth protocol and our strongest trained baseline.
One paper constrains the design. Deeper Insights into Weight Sharing in Neural Architecture Search reports ground-truth rank correlations as low as 0.45 for weight-sharing supernets. NAS-Bench-201 documents that differentiable search collapses to skip connections. We therefore do not build a shared supernet trained by predictive coding: its settling cost and interference problems would compound.
Research Program
We first build a ground-truth set. We will sample about 500 cells from the NAS-Bench-201 space: a 4-node cell with 6 edges, each edge choosing among zero, skip, 1×1 convolution, 3×3 convolution and 3×3 average pooling. Each cell is instantiated as a predictive coding graph. Rates pass through a rectifier with a learned positive prediction bias, errors are division mismatches, and feedback uses Kolen-Pollack weights, following the Tightening paper. We train at reduced width and epoch count on Fashion-MNIST and CIFAR-10. A random subset of 50 architectures is also trained at full budget, to confirm that the reduced-fidelity ranking holds, as the fixed-budget work claims it does for ordinary training. A second learning rule, random feedback weights, is run on a 150-architecture subset to test whether conclusions depend on the specific rule. Because NAS-Bench-201 already lists backpropagation accuracies for every cell, the backpropagation ranking costs nothing.
We then define the score. For a minibatch of $n$ examples at initialization, we clamp inputs and targets, run $K$ settling steps of the constrained rule, and record each example's weight update $u_i$ (concatenated across layers and normalized per layer). We form the kernel $K_{ij} = \cos(u_i, u_j)$ and score the architecture as
$$s = \log\det\left(K + \epsilon I\right) \cdot \mathbb{1}\left[\min_\ell \bar{r}_\ell > \tau\right],$$
where $\bar{r}_\ell$ is the mean firing rate in layer $\ell$ after settling, $\tau$ is a small collapse threshold, and $\epsilon$ is a small constant for numerical stability. The intuition is that an architecture whose rule produces nearly identical updates for different examples cannot learn to separate them. The indicator discards candidates where some layer has fallen silent. We will report the kernel and the collapse filter separately as well as together.
The analysis then asks whether predictive coding accuracy and backpropagation accuracy rank architectures differently. If they do, we ask which architectural features explain the difference. We will test operation counts, path depth from input to output, and the number of skip-only paths, which may carry errors unattenuated.
Baselines
The naive reference is random search over the space: sample architectures uniformly and train the top-scoring ones, which here means random ones. Every score must beat this under an equal number of trained candidates.
The strongest free competitor is backpropagation transfer. We rank architectures by their NAS-Bench-201 backpropagation accuracy and use that ranking for predictive coding. It is strong because the file's evidence shows predictive coding can match backpropagation accuracy on fixed networks. It costs nothing, so beating it requires a significantly higher Spearman correlation with predictive coding accuracy, established by bootstrap over architectures. No reported result was found on Lacuna for any ranking method evaluated against predictive-coding-trained accuracy.
The standard zero-cost scores are the next competitor: Synflow, SNIP, grad_norm and Jacobian covariance, their combination, and the input-separability score of Neural Architecture Search without Training. We compute each with backpropagation at initialization on the identical minibatch and architecture instances. On NAS-Bench-201 against backpropagation targets, the combined score reports 0.82 Spearman (Zero-Cost Proxies for Lightweight NAS). We also include the sparsity-aware Hamming variant from the spiking search paper, applied to rectified rates.
The strongest trained competitor is fixed-budget predictive coding training: a few epochs of the actual rule at reduced width, following Rethinking Performance Estimation. We match wall-clock time, not epochs. Our score must reach equal or higher rank correlation at the same wall-clock, or equal correlation at a fraction of it.
Two ablations isolate the key ingredient. In the first, the kernel is built from backpropagation gradients instead of the rule's own updates, so the kernel form is held fixed while the learning signal changes. In the second, the collapse filter is used alone. If the backpropagation-gradient kernel does as well as the rule-update kernel, the claim that the learning rule matters is false.
Why not just…?
Why not just rank by backpropagation accuracy from the tabular benchmark? This is the most serious threat, and it may win. If the two rules share their good architectures, the Spearman correlation between NAS-Bench-201 accuracy and predictive coding accuracy will be high, and any new score is pointless. We make this correlation the first measured result. If it exceeds about 0.9 on both datasets, we will report transfer as the answer and narrow the contribution to the collapse filter and a feature analysis of the residual disagreements. A reason to expect lower correlation is that operations like zero and skip change how division-mismatch errors propagate, and backpropagation accuracy does not see this.
Why not just use the existing zero-cost scores unchanged? Pruning scores evaluate backpropagation gradients, which are not the signals the network will learn from. In deep non-negative networks those gradients can vanish to $10^{-40}$ (Spiking Network Initialisation and Firing Rate Collapse). The spiking search paper already found forward-activity scores biased by zeros. Still, these scores might transfer tolerably, and the direct comparison, together with the backpropagation-kernel ablation, will show it.
Why not just train each candidate briefly with the real rule? Fixed-budget training is principled and is likely the strongest accuracy-per-candidate method. Its weakness is that every epoch includes $K$ settling steps, so its cost stays far above one settling pass. We compare the two on the same wall-clock axis. If short training dominates at every budget, the score is not useful.
Why not just train a weight-shared supernet with predictive coding? Shared-weight supernets already reach ground-truth correlation of only about 0.45 under backpropagation (Deeper Insights into Weight Sharing), and differentiable search collapses to skip connections on this space (NAS-Bench-201). Adding iterative settling to that setting adds cost without addressing either failure. We leave this out of scope.
Decisive Checks
The first check is the Spearman correlation between backpropagation and predictive coding rankings on Fashion-MNIST and CIFAR-10. This decides whether the problem exists.
The second check is the correlation of the rule-update kernel against every baseline, and against its backpropagation-gradient ablation. This decides whether the learning signal is what matters. We also plot top-10 regret as a function of wall-clock time for our score, fixed-budget training and random search. Here, regret is the gap between the best predictive coding accuracy among the ten architectures selected by a method and the best accuracy in the whole ground-truth set.
The third check is whether results replicate under the random-feedback rule. This decides whether the finding generalizes beyond one rule.
On CIFAR-10, a reference point for how far bio-constrained learning currently lags is 55.54% for counter-Hebbian learning (Biologically Inspired Learning Model for Instructed Vision), against 94.37% for the best backpropagation-trained cell in NAS-Bench-201. We will report where searched predictive coding architectures fall in that range, since part of that gap may reflect architecture choice rather than the learning rule.
References
- Tightening the Biological Constraints on Gradient-Based Predictive Coding
- Neural Architecture Search for Spiking Neural Networks
- Neural Architecture Search without Training
- Spiking Network Initialisation and Firing Rate Collapse
- Zero-Cost Proxies for Lightweight NAS
- Rethinking Performance Estimation in Neural Architecture Search
- Deeper Insights into Weight Sharing in Neural Architecture Search
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Biologically Inspired Learning Model for Instructed Vision