Placing Dendritic Units Without Training: A Covariance Test for Where Quadratic Neurons Pay Off

Real neurons do not just add up their inputs. Their branches multiply and gate inputs that arrive together. A recent line of work turns this "nonlinear dendritic integration" into a practical layer: a quadratic neuron that computes $x^T A x + w \cdot x + b$ instead of $\sigma(w \cdot x + b)$. Swapping a few convolutional layers for such neurons lifts ResNet-110 on CIFAR-100 from 70.84% to 72.40% with about 1% more parameters (Dendritic Integration Inspired Artificial Neural Networks Capture Data Correlation). The gain has two costs. The quadratic term raises the number of floating-point operations and slows training and inference. And the layers to replace were chosen by hand, by trial and error. Finding the right placement automatically is an architecture-search problem, and architecture search is expensive: judging candidates by training each one to convergence can take thousands of GPU hours.

We propose to decide placement from the reason quadratic neurons help in the first place. The same paper proves that quadratic neurons reach the optimal decision boundary when classes differ in their covariance rather than their mean. That suggests a cheap test. At each candidate layer, run a few minibatches through a freshly initialized (or one-epoch-warmed) network and measure how much of the class separation in that layer's inputs lives in second-order statistics that a linear unit cannot read. Then choose placements that maximize this "covariance advantage" under a compute budget. The claim is that a score derived from the mechanism picks better placements than generic "zero-cost proxies" (scores that rank architectures before training). Those proxies were built for convolution–batchnorm–ReLU blocks and are known to reward sheer model size.

Core Claim

Define $h_\ell$ as the input vectors a quadratic unit at slot $\ell$ would receive. For a pair of classes, fit Gaussians with shrinkage-regularized means $\mu_1, \mu_2$ and covariances $\Sigma_1, \Sigma_2$. Their divergence splits into a mean part and a covariance part:

$$ D_\ell = \underbrace{\tfrac12 (\mu_2-\mu_1)^T \Sigma_2^{-1}(\mu_2-\mu_1)}_{\text{linearly readable}} \;+\; \underbrace{\tfrac12\left[\operatorname{tr}(\Sigma_2^{-1}\Sigma_1) - d + \ln\tfrac{\det \Sigma_2}{\det \Sigma_1}\right]}_{C_\ell:\ \text{covariance-only}} $$

Here $d$ is the input dimension. The slot score $S_\ell$ is the covariance term $C_\ell$, averaged over sampled class pairs, divided by the mean term. It is high where classes are distinguishable mainly through feature co-variation.

Placement is then a small knapsack problem. Choose the set $P$ of slots that maximizes $\sum_{\ell \in P} S_\ell$ subject to $\sum_{\ell\in P} \mathrm{FLOPs}_\ell \le B$, where $B$ is the extra compute we allow.

The hypothesis is testable as stated. $S_\ell$, computed in seconds, should predict the trained accuracy gain from placing a quadratic unit at $\ell$. It should do so better than off-the-shelf proxies, and nearly as well as short training.

Relation to Prior Work

Two papers motivate the gap by showing the benefit without a way to find it.

  • Dendritic Integration Inspired ANNs (also Liu et al.) supply the quadratic unit and the covariance theory. They place layers manually, and their theory assumes Gaussian data. We reuse the unit and turn the theory into a placement score. We also test the score where features are far from Gaussian.
  • NSPDI-SNN adds quadratic integration to spiking neurons and discovers sparse structure by pruning during full training. That is structure search paid for with complete training runs, which is exactly the cost we aim to avoid.

A third paper constrains rather than supplies. Bounds on the computational complexity of neurons due to dendritic morphology shows on Boolean toy tasks that the number of compartments, not just their arrangement, sets how much a dendritic neuron can learn. We take this as a warning that placement and amount interact. The benchmark below therefore varies both, rather than assuming slot scores add up.

The search side supplies both methods and cautions:

  • Zero-Cost Proxies for Lightweight NAS and NAS-Bench-Suite-Zero give the evaluation protocol and the proxies we compare against. They also document two failures: proxies that break under unfamiliar topologies, and proxies that mostly track model size.
  • EZNAS evolves proxies automatically, but only over standard blocks.
  • DARTS offers a differentiable alternative. NAS-Bench-201 shows DARTS-style search collapsing onto parameter-free operations.

None of these scores a layer by what the data at that layer needs. Our score is built from class statistics, not weight or gradient magnitudes. It is therefore less exposed to the size bias, since a bigger quadratic unit does not raise $S_\ell$.

Research Program

We first build ground truth, because no reported result exists on Lacuna for any proxy's ranking quality on networks containing quadratic units.

ResNet placement benchmark. On CIFAR-100 with ResNet-56, we define nine candidate slots: the first, middle and last block of each of the three stages. Each slot can hold a channel-wise quadratic unit in the style of Liu et al. or not, giving 512 configurations. Each is trained with two seeds on a fixed schedule, for roughly a thousand GPU-hours. This lets us measure every placement's true accuracy and FLOPs, and check directly whether slot effects are additive.

NAS-Bench-201 extension. We add a quadratic 3×3 convolution as a sixth operation to the NAS-Bench-201 cell and sample 1,000 architectures from the enlarged space. These are trained under the original protocol on CIFAR-10, CIFAR-100 and ImageNet-16-120. This tests whether the score transfers to cell-level search.

Score computation. We compute $S_\ell$ at initialization and after one warm-up epoch. Features in a random network may carry too little class structure, and we will report honestly which variant is needed.

Non-Gaussian robustness. Because the score inherits the Gaussian assumption, we also build a distribution-free variant. It replaces the covariance term with a kernel two-sample statistic on centered second moments. We compare the two variants on layers whose features show heavy tails.

Transfer to ImageNet. Finally, the selected placement for ConvNeXt-B on ImageNet-1K is trained once and compared with Liu et al.'s hand-chosen placement.

Baselines

Do nothing, and the manual design. The naive reference is the network without quadratic units: ResNet-110 at 70.84% on CIFAR-100 and ConvNeXt-B at 83.8% top-1 on ImageNet-1K. The strongest existing design is Liu et al.'s hand placement, reported at 72.40% for Dit-ResNet-110 and 84.2% for Dit-ConvNeXt-B (Dendritic Integration Inspired ANNs). Beating it means matching or exceeding its accuracy at equal or lower added FLOPs, found with minutes of scoring instead of trial-and-error training runs.

Random placement at matched FLOPs. This checks whether any budget-matched placement works as well as a chosen one. We draw 20 random placements per budget.

Off-the-shelf zero-cost proxies. We run synflow, nwot, snip, grad_norm and FLOPs as scores through the same knapsack. We also run the Synflow–Jacob_cov–SNIP ensemble, the strongest reported proxy combination (Spearman ρ = 0.82 on standard NAS-Bench-201, Zero-Cost Proxies for Lightweight NAS). Two further proxy baselines complete the set. The first is EZNAS-A (Kendall τ = 0.65 on NAS-Bench-201 CIFAR-10, EZNAS). The second is an XGBoost surrogate fed all 13 proxies, which NAS-Bench-Suite-Zero reports improves rank correlation by up to 42%. Every proxy gets the same minibatches and the same number of forward and backward passes. The surrogate gets the same number of trained labels as any learned method. Those reported correlations come from standard spaces, not dendritic ones. The comparison that counts is therefore on our benchmarks, with a significant margin in Kendall τ and in top-10% recall.

Short training. Training each candidate for one to three epochs is the budgeted estimate from Rethinking Performance Estimation in NAS. For reference, the reduced-training method EcoNAS reached ρ = 0.61 on NAS-Bench-201 at roughly a thousand times proxy cost. We report ranking quality against wall-clock time for every method.

Differentiable relaxation. A DARTS-style supernet mixes linear and quadratic paths at each slot, trained under the same GPU budget as the other methods.

Mean-only ablation. Scoring slots by the mean (linearly readable) term alone isolates whether the covariance component is what carries the signal.

Why not just…?

Why not just widen or deepen ordinary layers with the same extra FLOPs? Liu et al. report that quadratic neurons beat two-layer perceptrons when classes differ only in covariance and in few-shot MNIST settings. They do not report a FLOP-matched comparison inside deep networks. We add one: every selected placement is paired with a linear network widened to the same FLOPs. If width matches the gain, the placement question is moot, and we will report that.

Why not just apply standard zero-cost proxies? Existing evidence says they struggle here. They fail across unfamiliar topologies (Zero-Cost Proxies for Lightweight NAS), assume standard blocks (EZNAS), and track size (NAS-Bench-Suite-Zero). That size bias would push them toward FLOP-heavy quadratic units. But nobody has measured this on dendritic spaces. If the proxy ensemble ranks placements as well as our score, we will adopt it and report the negative result.

Why not just train each candidate briefly, or use random search? This is the real threat. NAS-Bench-201 shows simple random and evolutionary search are strong, and one-epoch training may rank placements well. Our score's only advantage then is cost, and our warm-up variant partly concedes the point. The benchmark settles it by plotting ranking quality against wall-clock time. If short training dominates at every budget, the mechanistic score is not worth using.

Why not just relax placement with DARTS? The documented collapse onto parameter-free operations suggests the supernet may drift toward whichever path trains fastest early. Quadratic paths have input-dependent gradients, $2Ax + w$, so they may behave differently from linear ones. We measure how often the supernet's chosen placement lands in the top decile of the exhaustive table.

Decisive Checks

On the 512-configuration ResNet table, the covariance score must beat both the mean-only ablation and FLOPs-as-score in Kendall τ with true accuracy gain. If it does not, the core claim fails.

The knapsack placement must reach at least Liu et al.'s accuracy at no more added FLOPs.

Either the initialization or the one-epoch variant must sit on the accuracy-versus-wall-clock frontier alongside short training, not beneath it.

On the NAS-Bench-201 extension, the score's ranking quality must hold on ImageNet-16-120, not only on CIFAR.

On non-Gaussian layers, the kernel variant should recover whatever ranking quality the Gaussian score loses. If it does not, the covariance theory is a poor guide beyond its assumptions.