Correlation-Aware Relevance Pruning for Densely Connected Networks

A DenseNet-style network passes every layer's output forward by concatenation, so each later layer sees all earlier features. This is useful, but it is wasteful. Log-DenseNet: How to Sparsify a DenseNet removed most of the cross-layer connections from such a network without losing segmentation accuracy, which shows that much of the concatenated stack is redundant. The natural fix is to learn which channels each layer needs, and Bayesian relevance pruning does this. In that approach, each channel gets a scalar gate with its own prior variance, and channels whose variance collapses to zero are removed. This is automatic relevance determination (ARD): a classical technique that sets one prior variance per input or feature by maximizing the evidence, so irrelevant features shrink out on their own. The existing deep version, Variational Convolutional Neural Network Pruning, treats all gates as independent. Dense concatenation is exactly the setting where that assumption fails. When channel A from layer 2 and channel B from layer 5 carry the same information, the loss is flat along "use A or use B" and sharp along "use neither." An independent posterior can only describe this by giving each gate a small variance. Both channels then look precisely needed, and both are kept.

We propose to put a low-rank-plus-diagonal Gaussian posterior over the gates of the concatenated feature stack, place ARD priors both on individual gates and on the columns of the low-rank factor, and prune with a Bayesian version of the Optimal-Brain-Surgeon rule. That rule scores each gate by its marginal rather than conditional uncertainty, and it moves the surviving gates to compensate for each removal. The claim is specific. In densely connected networks, redundancy appears as strong negative posterior correlation between gates. A rank-$r$ covariance with $r$ on the order of tens is enough to expose it. Exposing it lets one-shot pruning remove substantially more channels and connections, at equal accuracy, than independent-gate pruning or fixed sparsity patterns.

Core Claim

Let $h_j$ be the $j$-th entry of the concatenated input to some layer, indexed by (consumer layer, source channel). The layer reads $z_j h_j$ in place of $h_j$. We place the prior $z_j \sim \mathcal{N}(1, \alpha_j^{-1})$ around the identity gate. We fit the variational posterior $q(z) = \mathcal{N}(\mu, \Sigma)$ with $\Sigma = D + VV^\top$, where $D$ is diagonal and $V$ has $r$ columns. This follows the structure used by Efficient Low Rank Gaussian Variational Inference for Neural Networks and SLANG, but here it is applied to a few thousand gates instead of millions of weights. Because the gates live in a small space, we sample them directly as $z = \mu + D^{1/2}\epsilon_1 + V\epsilon_2$ at cost $O(dr)$. We do not need a local reparameterization approximation, which is the step that broke down for convolutions in the low-rank weight-space work.

When the posterior covariance approximates inverse curvature, setting gate $j$ to zero and letting the others readjust raises the loss by about

$$\Delta_j = \frac{\mu_j^2}{2\,\Sigma_{jj}}, \qquad \delta\mu = -\frac{\mu_j}{\Sigma_{jj}}\,\Sigma_{:,j}.$$

Here $\Sigma_{jj} = D_{jj} + \lVert V_{j,:}\rVert^2$ is the marginal posterior variance of gate $j$, and $\delta\mu$ is the compensating shift applied to all other gate means. Both formulas are the standard second-order pruning step, with the posterior covariance standing in for the inverse Hessian. Under a mean-field posterior, $\Sigma_{jj} = D_{jj}$ is close to the conditional variance and $\delta\mu$ is zero. Redundant channels therefore receive inflated scores, and removing one gets no compensation. This is the same "trend reversal" SLANG documented for mean-field variances, and here it maps directly onto a pruning error. With the low-rank term, the shared axis of a substitutable pair appears in $\lVert V_{j,:}\rVert^2$. One channel of the pair is then cheap to remove, and $\delta\mu$ moves its weight onto the partner. Woodbury identities keep all of this linear in the number of gates.

Research Program

The work has three parts.

The first part fixes the optimization tension between ARD and low-rank variational inference. ARD needs variances that move freely toward zero. ELRG-VI found that learning the posterior diagonal $D$ made deep networks underfit, and that fixing it to a small constant was needed for stable training. Joint Model and Data Sparsification via the Marginal Likelihood likewise needed damping and warm starts to stop per-parameter variances from "explaining away" signal. Our response is to separate the two roles. The posterior diagonal $D$ stays fixed and small, as in ELRG-VI. All relevance shrinkage happens in the prior precisions $\alpha_j$, which are updated in closed form by the usual EM step $\alpha_j = 1/((\mu_j-1)^2 + \Sigma_{jj})$ every few epochs, with damping. We test directly whether this separation avoids underfitting, and whether a learned-$D$ variant reproduces the ELRG-D-VI failure.

The second part chooses the rank automatically. ELRG-VI and Stochastic Segmentation Networks both name the hand-set rank as an open problem. Scalable Bayesian GPFA with automatic relevance determination showed, in a linear factor model, that ARD priors on loading columns recover intrinsic dimension. We reuse that method on the columns of $V$, starting with a generous $r$ such as 64 and letting unused redundancy modes shrink away. The surviving rank is itself a diagnostic: it estimates how many independent families of substitutable features a dense block contains.

The third part is the pruning procedure. After training, we score and remove gates greedily, recomputing $\Sigma$ by rank-one downdates and applying $\delta\mu$ after each removal. We stop at a target budget or at a threshold on the summed $\Delta$. A source channel whose gates are zero for every consumer never needs to be stored. This attacks the activation-memory cost that Log-DenseNet identifies as the practical bottleneck of dense networks. We do not fine-tune after pruning, because the compensation step is meant to replace fine-tuning. We will check whether that holds.

Baselines

The naive reference point is the unpruned dense network: DenseNet-BC on CIFAR-10 and CIFAR-100, and FC-DenseNet-103 on CamVid. Alongside it, we run uniform random removal of connections at matched budgets. This tells us how much of the reported compression is free under any method, which Log-DenseNet's results suggest may be a lot.

The fixed-pattern baseline is Log-DenseNet, which connects layer $i$ only to layers $i-2^k$. It is the strongest competitor on CamVid, where its paper reports 67.3% mean IoU for Log-DenseNet-103 with 50% fewer parameters, against 66.9% for FC-DenseNet-103. We will retrain it with our training recipe and epoch budget. To beat it, our method must match or exceed its mean IoU at no more parameters and lower peak activation memory, or reach clearly fewer connections at equal mean IoU.

The strongest learned baseline is mean-field variational channel pruning (Variational Convolutional Neural Network Pruning). That paper reports a 74% channel reduction on CIFAR-10 ResNet-164 with a 0.42% accuracy drop, and 40% pruning of ResNet-50 on ImageNet at 75.2% versus 75.1% top-1. No reported result was found on Lacuna for its DenseNet runs, so we will reproduce it on our DenseNet backbones. We will give it the same gate placement, the same epochs, and an equal hyperparameter search, roughly twenty configurations each. Beating it means a higher accuracy-versus-remaining-channels curve without fine-tuning at the 50–80% pruning range on CIFAR-100, where redundancy should matter most.

The key ablations isolate one ingredient each. The first runs our exact code with $r=0$, which tests the correlation term. The second keeps $r>0$ but scores gates with $D_{jj}$ only and skips $\delta\mu$, which tests whether the benefit comes from the pruning rule or merely from training with a correlated posterior. The third fixes $r$ without ARD on the columns of $V$, which tests automatic rank. The fourth learns $D$, which tests the stability fix. For calibration, the only related figure in the file is ELRG-VI's ECE of 0.018 versus 0.071 for mean-field on ResNet-18 CIFAR-10. That setting is a different one, so we cite it only as a sanity reference and do not claim to beat it.

Why not just…?

Why not just use mean-field variational pruning, which already works on DenseNet? This is the main threat. If dense blocks contain few truly substitutable channel pairs, independent gates may be good enough. Our argument against it is mechanistic: a mean-field posterior reports conditional variance, so it must overrate redundant channels. That argument is only worth something if it shows up in the numbers. The planted-duplicate experiment below and the $r=0$ ablation settle it. If the curves coincide on CIFAR-100 and CamVid, we will report that dense redundancy is not of the substitutable kind.

Why not just cluster channels by activation correlation and keep one per cluster? This is cheap, deterministic, and a fair challenge. Correlated activations are not the same as interchangeable roles in the loss. Two channels can co-activate while downstream layers use them differently, and two uncorrelated channels can be interchangeable once later layers mix them. We include this as a baseline at matched budgets, with greedy removal by activation-correlation clusters plus a least-squares refit of the consuming $1\times1$ convolution. If it matches our method, the Bayesian machinery is unnecessary, and we will say so.

Why not just use Log-DenseNet's fixed pattern? It needs no training-time cost and does well on CamVid. But it is the same for every dataset and every layer. If our learned connection masks fall near powers of two, then Log-DenseNet already captured the structure. If they differ systematically across tasks, for example keeping more long-range links for segmentation than for classification, then data-driven pruning adds something. We will compare the masks directly.

Proposed Experiments

The decisive first test is a controlled one. Inside a trained DenseNet-BC block, we append exact or noise-perturbed copies of selected channels to the concatenation, then retrain the gates. A method that understands redundancy should keep one copy of each, show strong negative posterior correlation between copies, and show no accuracy loss when the duplicate is pruned. Mean-field gating should keep both copies or weaken both. This test also checks whether the ARD-selected rank tracks the number of planted duplicate families.

The main experiments use CIFAR-10 and CIFAR-100 with DenseNet-BC, and CamVid with FC-DenseNet-103. We report accuracy or mean IoU against remaining channels, connections, FLOPs, and measured peak training-activation memory, at budgets from 30% to 90% removal and without fine-tuning. A second table repeats the comparison with a short fine-tuning phase, to measure how much of the gap the compensation step $\delta\mu$ closes. If time allows, an ImageNet DenseNet-121 run tests scale. Its baseline is the mean-field pruning paper's ImageNet result on ResNet-50, which uses a different backbone and so is not directly comparable.

Beyond accuracy, we measure how stable the gates are across seeds, using the overlap of kept sets. We expect correlation-aware pruning to give more consistent choices within redundancy groups, while still choosing arbitrarily between the members of each group. We also track per-iteration overhead, which should stay under a few percent because the gate dimension is small.

The proposal fails cleanly in any of three ways: the $r=0$ ablation matches the full method, the activation-clustering baseline matches it, or learned masks reproduce Log-DenseNet's pattern. Each outcome would tell us something definite about where the redundancy in dense networks actually lives.