Paying for Distance: Offset-Resolved Rademacher Penalties for Global-Kernel CNNs

Convolutional networks that see the whole image in one layer are now competitive with Transformers. One example is ParCNetV2, which uses one-dimensional kernels about twice as long as the feature map. Two things come with that global view. Every channel gains many extra weights at long spatial offsets. Because the kernel is zero-padded, those far weights touch only a few output positions near the border, so they behave like position-specific parameters. The far weights are therefore the part of the model most able to memorize noise, and these networks are trained with the same generic weight decay and stochastic depth as small-kernel models. We propose to measure, during training and for each band of kernel offsets, how much that band helps the network fit random labels. We then penalize each band's weights in proportion to that measurement. The network keeps long-range weights where they carry signal and shrinks them where they only add capacity to fit noise.

The tool for this measurement is Rademacher complexity: the expected best correlation a model class can achieve with random ±1 labels. It is the standard way to state how much a model can overfit. Earlier work has turned Rademacher bounds into training penalties, but always per neuron or per layer, never per spatial distance. Earlier work on long-range vision models, for its part, has never controlled capacity by distance. The result we test is a regularizer that, in effect, learns how far each layer and channel should look, chosen by a generalization argument rather than by a hand-set kernel size or a fixed decay profile.

Core Claim

The extra capacity of a global-receptive-field convolution is concentrated in a small number of offset bands, and that concentration depends on the data. The location of those bands differs across layers, across channels, and across amounts of training data. A penalty that estimates per-band noise-fitting capacity from mini-batches should therefore beat two kinds of alternatives in the low-data regime where global kernels overfit. The first is uniform regularizers such as weight decay and DropBlock. The second is fixed distance-decay priors of matched strength. The advantage should shrink as training data grows. If a tuned fixed distance-decay prior matches it, the data-dependent ingredient adds nothing and the claim fails. We treat that comparison as the central test.

Prior Work and Positioning

Three papers motivate the approach and supply its method. Adaptive Dropout with Rademacher Complexity Regularization bounds Rademacher complexity in terms of per-neuron keep rates and optimizes those rates. Its visualizations show that the learned rates spontaneously keep central pixels and drop the periphery. That is evidence that a capacity argument already "wants" spatial structure, but the effect there is an emergent side effect, never an explicit control over distance. Beyond Dropout: Feature Map Distortion learns feature perturbations that lower empirical Rademacher complexity, using a first-order look-ahead through the next layer. We borrow its idea of a cheap gradient-based estimate of the complexity term, but we apply it to kernel offsets rather than to activations. An Empirical Study on Regularization by Local Rademacher Complexity shows that a local Rademacher penalty can be computed by sampling random signs per mini-batch, with little overhead, and that it combines well with mixup. That result is the main reason we expect the estimator to be affordable.

One paper constrains the design. LocalDrop controls local Rademacher complexity through keep rates and periodic singular value thresholding. Singular value thresholding is a projection step that zeroes small singular values of a weight matrix to force it toward low rank. It costs 40–60 minutes per 100 ImageNet epochs, and generic low-rank pressure applied to an oversized kernel would act on all offsets at once. We avoid both problems: we use no decomposition, and we penalize each band separately.

On the receptive-field side, Understanding the Effective Receptive Field shows that the effective receptive field (the input-gradient map $\partial y/\partial x$) of a deep CNN is roughly Gaussian and grows only as the square root of depth. That paper only diagnoses the effect; we use its measurement as an outcome variable. ParCNetV2 and CCNet obtain a global receptive field through oversized kernels and criss-cross attention respectively, and both use heuristic training only. We take ParCNetV2's oversized depthwise kernel as the main testbed because its parameters are tied to explicit spatial offsets, which is where an offset-resolved penalty is cleanest. CCNet is a secondary transfer target.

Research Program

Each oversized depthwise kernel $w_c$ for channel $c$ is split into distance bands $b$, for example offsets 0–3, 4–7, 8–15, and so on up to the full kernel length. On every $k$-th training step we draw random class labels for the mini-batch and compute the gradient of the network's cross-entropy on those random labels with respect to each band's weights. A gradient on random labels measures how much moving that band would improve the fit to noise. This is a first-order, local estimate of the empirical Rademacher complexity contributed by the band. Its moving-average norm is $g_{c,b}$. The training loss becomes

$$ L = L_{\text{task}} + \lambda \sum_{c}\sum_{b} g_{c,b}\,\lVert w_{c,b}\rVert_2 , $$

where $L_{\text{task}}$ is the usual cross-entropy, $w_{c,b}$ is the slice of channel $c$'s kernel in band $b$, and $\lambda$ is a single global strength. The term is a group-lasso penalty, meaning one that can shrink a whole band to zero together, whose per-group price is set by measured noise-fitting capacity rather than by a prior. Each refresh costs one extra backward pass. With $k \approx 10$ we expect the overhead to stay under about 10%, and we will report wall-clock time against LocalDrop's stated cost.

The theoretical part is deliberately modest. For a single depthwise layer followed by a Lipschitz head, we will derive a bound in which each band's contribution scales with its weight norm times the energy of the inputs it actually reaches. Under zero padding, that energy differs from band to band. The bound justifies the form of the penalty, and $g_{c,b}$ serves as its practical, nonlinearity-aware stand-in. We will not claim the estimate is a valid bound for deep networks.

The analysis has three parts. We record the learned band-norm profile per layer and per channel. We measure the effective receptive field with the gradient method of Luo et al. and test whether the penalty yields layer-specific, non-Gaussian profiles that a fixed prior could not produce. We also check whether $g_{c,b}$ is largest for the far, border-only offsets that carry ParCNetV2's implicit positional encoding.

Baselines

The naive reference point is ParCNetV2 trained with its published recipe: weight decay and stochastic depth, with no added regularizer. The strongest reported result for this architecture is ParCNetV2-B at 84.6% top-1 on ImageNet-1K with 56M parameters, and ParCNetV2-T at 52.6 box AP on COCO with Cascade Mask R-CNN, both from ParCNetV2. We found no reported result on Lacuna for ParCNetV2-T top-1 on ImageNet, for ParCNetV2 on CIFAR-100, or for ParCNetV2 under reduced training data, so we will reproduce those settings ourselves.

The strongest existing complexity-derived regularizers are Block Feature Map Distortion and LocalDrop. The file reports Block FMD at 78.76% top-1 on ResNet-50 ImageNet against DropBlock's 78.13% (Beyond Dropout), and LocalDrop at 21.1% error against DropBlock's 22.2% (LocalDrop). These are the methods most likely to win on regularization alone. We will port both, together with DropBlock, onto the ParCNetV2 backbone.

The critical ablation isolates the key ingredient, the data-dependent price $g_{c,b}$. Its first variant replaces $g_{c,b}$ with a fixed monotone distance profile, either linear or Gaussian in offset, with the shape and strength tuned on validation data. Its second variant uses a uniform price, which reduces the penalty to plain group lasso. A third, signal-normalized variant divides $g_{c,b}$ by the band's gradient norm on the true labels; it tests whether a noise-to-signal ratio outperforms the raw complexity estimate.

To keep comparisons fair, every method gets the same data, schedule, and augmentation. Each method's regularization strength gets the same tuning budget of eight trials on a held-out split, and results are averaged over three seeds. Beating the field means two things. First, the method must exceed both the tuned fixed-distance prior and the best of Block FMD and LocalDrop by more than the seed spread at the 10% and 25% data fractions. Second, it must be no worse than the published recipe at full data.

Why not just…?

Why not just use DropBlock or spatial dropout? These methods drop contiguous regions at random, which regularizes local redundancy but is blind to which offsets are fitting noise. Prior comparisons in the file already show complexity-derived methods ahead of DropBlock on ResNet-50, at 78.76% versus 78.13% for Block FMD. That evidence comes from small-kernel networks, though. DropBlock is included as a tuned baseline, and if it matches us on global-kernel models, the distance resolution is not doing work.

Why not just add a fixed distance-decay penalty, or shorten the kernel? This is the real threat. If long-range overfitting is uniform across layers and datasets, a single hand-tuned decay curve captures everything our estimator measures, at zero overhead. The fixed-profile ablation decides the question directly. We also report the learned profiles: if they come out as the same monotone curve in every layer, we will say the simpler prior suffices. Shortening the kernel discards the global view that gives ParCNetV2 its accuracy, and it is covered by the far-band-zeroed variant within the same ablation.

Why not just rely on the local branch that ParCNetV2 already has? ParCNetV2 pairs a 7×7 local convolution with the global kernel. However, Luo et al. show that parallel short paths and skip connections shrink the effective receptive field without limiting what the wide branch can memorize. Since all our runs keep the local branch, any gain we report comes on top of it.

Why not just use mixup and stronger weight decay? The local Rademacher study found its penalty complementary to mixup. We will run every method with and without mixup to check that the same holds for our penalty.

Proposed Experiments

The main experiment trains ParCNetV2-T on ImageNet-1K at 10%, 25%, and 100% of the training data, and on CIFAR-100 at full data. The prediction is that our gain over the tuned fixed-distance prior grows as data shrinks, because the complexity estimate matters most where variance dominates.

The second experiment is a memorization test on CIFAR-100 with 40% of labels randomly corrupted. We measure training accuracy on the corrupted subset and the band-wise norms of the kernels. The penalty should cut memorization mainly by shrinking far bands, which is the mechanism the core claim asserts.

For transfer, we fine-tune the regularized backbones with Cascade Mask R-CNN on COCO and compare against the 52.6 box AP reference. We also run ADE20K segmentation; the file reports no ParCNetV2 number there, while CCNet reports 45.76% validation mIoU on it. As an extension, we apply the band penalty to distance-binned output gates in CCNet's criss-cross attention on Cityscapes with 25% of the training images, using CCNet's 81.9% test mIoU at full data as context.

The decisive checks are the following. The method must beat the tuned fixed-distance prior at low data. Its learned profiles must differ measurably across layers. Its overhead must be well below LocalDrop's 40–60 minutes per 100 epochs. If the first check fails, we will report that distance-aware weight decay is the practical answer and that the Rademacher machinery is unnecessary here.