Position-Calibrated Sparsity for Large-Kernel ConvNets

Convolutional networks see the image through small windows, so to connect distant regions they either stack many layers or use very large filters. The very large filters (51×51 and beyond) only became affordable once researchers made them sparse, meaning most of their weights are forced to zero. Sparsifying a network with a penalty term is notoriously fragile, though. The strength of the penalty has a narrow usable range: slightly too weak and nothing is pruned, slightly too strong and the network collapses. This note argues that, inside a large filter, these two problems are one problem. The training signal a filter weight receives shrinks sharply with its distance from the filter's center. A single penalty strength applied to every position therefore removes the outer ring of the filter first. A sparse "large-kernel" network can quietly turn back into a small-kernel network while its classification accuracy barely moves.

We propose to measure this effect directly and to remove it with a simple fix. The fix scales each filter position's penalty by a running estimate of the training signal that position actually receives. Pruning then competes weights against others under comparable gradient pressure, rather than against the filter's geometry. The claim is concrete and testable. With a uniform penalty, the weights kept at each radius fall from the outside in as the penalty grows, and the damage shows up on dense prediction (segmentation) long before it shows up on ImageNet accuracy. With a position-calibrated penalty, the kept weights spread evenly across radii, the useful penalty range widens, and segmentation quality holds at matched sparsity.

Core Claim

Understanding the Effective Receptive Field in Deep Convolutional Neural Networks showed that the effective receptive field is roughly Gaussian. This is the region of input pixels that actually influences an output, measured by the gradient of a central output with respect to each input pixel. Its width grows only like the square root of depth, and residual connections make it narrower still. The same decay applies to gradients reaching individual kernel offsets in a filter. Peripheral weights of a 51-wide strip receive systematically smaller task gradients than central ones.

The sparsity methods in the file penalize every parameter identically, regardless of its position in the kernel. These are hard-concrete gating in Learning Sparse Neural Networks through L_0 Regularization, thresholded masks with a straight-through estimator in Effective Neural Network L0 Regularization With BinMask, and batch-norm scale penalties in Learning Efficient Convolutional Networks through Network Slimming. A uniform penalty is a constant force pulling against a gradient whose size depends on position. So the penalty strength at which a position dies is position-dependent. We hypothesize that much of the reported "razor's edge" behavior in large kernels is this geometric bias, followed by a final collapse of the center.

The fix replaces the uniform penalty with a position-weighted one:

$$\mathcal{L} = \mathcal{L}_{\text{data}} + \lambda \sum_{l} \sum_{(u,v)} \frac{\bar{s}_l}{s_l(u,v)} \sum_{c} q(z_{l,c,u,v} \neq 0)$$

The symbols are defined as follows:

  • $l$ indexes layers, $(u,v)$ the offset within a kernel, and $c$ the channel.
  • $q(z \neq 0)$ is the probability that a weight's gate is open (the expected L0 count of Louizos et al.).
  • $s_l(u,v)$ is an exponential moving average of the mean absolute task gradient reaching offset $(u,v)$ in layer $l$, pooled over channels.
  • $\bar{s}_l$ is its mean over offsets.

Within a position, weights still compete on their individual merit. Across positions, the systematic radial decay is divided out. A global sparsity target is enforced by adjusting $\lambda$ with a dual-ascent controller, so the method is compared to baselines at the same sparsity rather than at the same penalty value.

Relation to Prior Work

The closest work is More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity (SLaK). SLaK decomposes an M×M kernel into M×5 and 5×M strips and trains them with magnitude-based prune-and-grow at a fixed target sparsity. That paper supplies our architecture and is our main competitor. It sidesteps penalty sensitivity by fixing sparsity directly. However, its magnitude criterion treats outer and central weights alike, so it inherits whatever magnitude gradient the radial gradient decay induces. Whether that hurts SLaK is an empirical question we test rather than assume.

Louizos et al. and BinMask supply the gating machinery. Both were evaluated only on small-kernel networks (LeNet, Wide ResNet, ResNet32/VGG19), and neither conditions the penalty on spatial position.

Network Slimming documents the λ sensitivity that motivates the work, as do Soft Weight-Sharing for Neural Network Compression and BayesNAS. Network Slimming prunes whole channels, so it cannot address spatial extent.

Luo et al. motivates the work twice: once through the Gaussian decay itself, and once through their peripheral-heavy initialization. They note that training pulls this initialization back toward a central profile. Our method is a training-time counterpart to that initialization.

Non-kernel routes to global context are outside our scope. These include criss-cross attention in CCNet, deformable sampling in DAS, and oversized kernels in ParCNetV2. ParCNetV2 is a natural second architecture if the effect holds.

Research Program

The first stage is diagnostic and could stand alone as a finding. We train SLaK-T-style decomposed-kernel networks under uniform hard-concrete and uniform BinMask penalties across a log-spaced sweep of λ. At each λ we record four quantities:

  • ImageNet top-1 accuracy;
  • the fraction of retained weights as a function of distance from the kernel center;
  • the measured effective receptive field, using Luo et al.'s gradient method;
  • the per-offset gradient statistic $s_l(u,v)$.

The hypothesis predicts three things. First, retention collapses from the periphery inward. Second, the λ at which each offset dies tracks its $s_l(u,v)$. Third, the effective receptive field shrinks toward that of a small-kernel network well before top-1 accuracy drops.

The second stage introduces the position-calibrated penalty and repeats the sweep. We then define the usable window as the width, in log λ, of the range over which two conditions both hold: sparsity reaches the target, and accuracy stays within one point of the dense model. This width is reported per method. The third stage takes the best configuration of each method to a full training schedule and transfers it to segmentation.

Baselines

The naive reference points are two dense models trained with the identical recipe: the dense decomposed-kernel network, and a dense small-kernel network of matched FLOPs. Together they bound what sparsity costs and what large kernels buy.

The strongest existing method is SLaK's own prune-and-grow at its published target sparsity. It is the strongest competitor because it avoids penalty tuning entirely and has strong reported numbers. SLaK-T reaches 82.5% ImageNet-1K top-1 according to SLaK, and SLaK-B reaches 50.2 mIoU on ADE20K in the same paper. No reported result was found on Lacuna for SLaK-T on ADE20K or for SLaK on Cityscapes, so we will produce those numbers ourselves with the released recipe. We will rerun SLaK under our exact data, epochs and augmentation rather than compare to the published figure alone.

The penalty baselines are uniform hard-concrete L0 and uniform BinMask. Each gets the same dual-ascent sparsity controller and the same number of tuning runs as our method. All methods are compared at identical global sparsity and identical training compute. A post-hoc magnitude-pruning-plus-fine-tuning baseline gets the same total epochs.

Two ablations isolate the key ingredient. The first is uniform penalty with the controller. It shows whether controlling global sparsity alone fixes the problem, or whether spatial allocation matters. The second is an analytic inverse-Gaussian weighting, which uses a fixed radial curve from Luo et al.'s theory instead of measured gradients. It shows whether measuring the per-offset gradient matters.

Beating SLaK means equal or better ImageNet top-1, and strictly better ADE20K and Cityscapes mIoU, at the same sparsity and compute. Beating the uniform-penalty baselines means a usable window at least twice as wide in log λ, plus flat radial retention.

Why not just…?

Why not just use SLaK's prune-and-grow, which never touches λ? This is the real threat. If SLaK's magnitude criterion already keeps peripheral weights, our method's advantage reduces to convenience. The stage-one radial retention measurement, run on SLaK as well, settles it. If SLaK's retention is flat across radius and its effective receptive field matches the dense model, we will report that the bias is specific to penalty-based sparsity. In that case the contribution narrows to making penalty methods usable for large kernels. If SLaK's retention also decays with radius, a radius-normalized magnitude criterion becomes a direct follow-up.

Why not just fix global sparsity with a Lagrangian controller and drop the spatial weighting? A controller chooses how many weights survive, not which ones. Under a uniform penalty the controller would still spend the pruning budget on the periphery. The uniform-penalty-plus-controller ablation tests exactly this. If it matches our method on radial retention and segmentation, the spatial weighting is unnecessary and we will say so.

Why not just exempt the outer positions from the penalty, or use dilated convolutions? In M×5 strips with M=51, positions far from the center hold most of the parameters. Exempting them forfeits most of the compression. We will include an exempt-outer-ring variant to quantify this. Dilated convolution is a different architecture that skips intermediate pixels. The file's dilated baselines sit far below large-kernel models on dense tasks: 67.1% Cityscapes mIoU for Dilated10 in The Cityscapes Dataset, and 32.31% ADE20K mIoU for DilatedNet in Scene Parsing through ADE20K Dataset, against SLaK-B's 50.2. These are older models trained under different recipes, so the gap is suggestive rather than a controlled comparison. We therefore treat dilation as context, not as a competitor to sparse large kernels.

Proposed Experiments

The λ sweeps run on ImageNet-1K with a shortened schedule, at seven λ values per method and three seeds. The best configuration per method then gets the full SLaK-T schedule. Transfer is to ADE20K with UperNet, following SLaK's segmentation setup, and to Cityscapes, where high resolution makes long-range context matter most. The metrics are:

  • top-1 accuracy and mIoU;
  • usable-window width in log λ;
  • retained-weight fraction by kernel radius;
  • effective-receptive-field area at 50% gradient mass;
  • wall-clock throughput, since sparse large kernels already need custom kernels.

Three outcomes are decisive. The premise fails if, under a uniform penalty, the outer ring is not pruned disproportionately or the effective receptive field does not shrink faster than top-1 accuracy falls. In that case the project stops at the diagnostic. The method succeeds if position calibration flattens radial retention and widens the usable window, and the gradient-measured weighting beats the analytic Gaussian one. It matters in practice only if, at matched sparsity, it closes or reverses the segmentation gap to SLaK while matching SLaK's 82.5% ImageNet top-1 under our rerun.