Reach or Routing? Separating Receptive Field Size from Attention in Catastrophic Forgetting

When a vision network is trained on one data distribution and then another, it tends to overwrite what it learned first. This is catastrophic forgetting. A striking result from federated learning, Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning, found that ordinary convolutional networks collapse on earlier clients when a model is passed sequentially from client to client, while Vision Transformers (image models built on self-attention, which lets every patch look at every other patch) keep their knowledge. The authors credit global context. But a Transformer differs from a ResNet in several ways at once. Its receptive field (the region of the image that can influence one output feature) is global. It routes information in a data-dependent way through attention. It uses a different normalization, and it has a different macro-architecture. Large-kernel convolutional networks such as RepLKNet and SLaK now reach near-global receptive fields with fixed, input-independent filters. That makes the confound separable for the first time.

We propose a controlled factorial study inside one backbone. The spatial mixing operator is the only thing that changes, and it is crossed along two axes: how far it reaches (local vs. near-global effective receptive field) and how it mixes (static convolution vs. attention). We then measure forgetting under sequential distribution shift. The claim to test is concrete. Under domain-style shift, where the label space stays fixed but local image statistics change, forgetting should fall with measured receptive field size roughly independently of whether attention is used. Under class-incremental shift, where new labels arrive, receptive field size should matter little. If the claim holds, practitioners can retrofit forgetting resistance into efficient convolutional backbones without paying attention's quadratic cost. If it fails, we will have shown that data-dependent routing, not reach, is the operative ingredient.

Core Claim and What Is New

Prior work either compares small-kernel CNNs to Transformers in sequential training (Rethinking Architecture Design…), or studies large-kernel CNNs only on static robustness benchmarks. The latter is Revealing the Dark Secrets of Extremely Large Kernel ConvNets on Robustness, which shows RepLKNet gains shape bias, resilience across the frequency spectrum, and resistance to layer removal, and that swapping its 31×31 kernels for 3×3 destroys these properties. Continual Transformer methods, Continual Learning with Lifelong Vision Transformer and Online Continual Learning with Contrastive Vision Transformer, add external attention keys and class anchors on top of attention. They never ask whether attention itself is needed.

Across the file, nobody has trained large-kernel convolutional networks sequentially. Nobody has crossed reach and routing so that each factor can be isolated. That crossing is the contribution.

Our hypothesis has a mechanism. Small receptive fields bind features to local texture and lighting statistics. When a new client or domain changes those statistics, gradients on the new data push shared early-layer weights in directions that conflict with the old task. A wide receptive field pools over enough of the image that the features become shape-like, as the robustness study found. Shape is shared across domains, so gradients from new domains should align better with old ones.

This predicts two things. First, forgetting should track the gradient-measured effective receptive field, the measure introduced in Scaling Up Your Kernels to 31×31, and not nominal kernel size. Second, the benefit should shrink when the shift is semantic rather than textural.

Research Program

All variants share one ConvNeXt/RepLKNet-style macro-architecture: residual blocks, depthwise spatial mixing, and pointwise channel mixing. We fix depth and stage widths and match parameters and FLOPs to within about 10% by adjusting width. The spatial mixer takes one of the following forms:

  • depthwise 3×3 and 7×7 convolutions;
  • a RepLKNet-style 31×31 depthwise kernel with a parallel small kernel merged at inference, the re-parameterization trick that makes large kernels trainable;
  • SLaK's decomposed 51×5 plus 5×51 sparse kernel from More ConvNets in the 2020s;
  • WTConv's wavelet-based mixer, using the fast implementation of Fast and Memory-Efficient Wavelet Convolutions;
  • a HyenaPixel global implicit filter from HyenaPixel;
  • a fixed dilated 3×3 stack whose nominal receptive field matches the 31×31 kernel;
  • two attention mixers: global self-attention, and non-shifted windowed attention with a 7×7 window.

The windowed-attention and dilated arms supply the off-diagonal cells of the factorial. Windowed attention has routing without reach. The dilated stack has nominal reach with sparse coverage, and Dilated convolution with learnable spacings documents that such gridded sampling skips pixels.

Normalization is crossed as a second factor, BatchNorm vs. LayerNorm, on the 7×7, 31×31, and global-attention arms. Otherwise normalization could explain the original CNN-versus-Transformer gap on its own.

For every variant we measure four things after training on each task:

  • the effective receptive field radius;
  • accuracy on all earlier tasks;
  • per-layer representation drift between checkpoints;
  • cosine similarity between the gradient of the current task loss and the gradient of the stored-task loss.

This lets us regress forgetting on measured receptive field, conditioned on mixer type, rather than just ranking architectures.

Two training regimes are run. In the from-scratch regime, the network sees only the continual stream. In the pretrained regime, every variant is pretrained with an identical ImageNet-100 recipe before the stream. Robustness properties of large kernels were reported for pretrained models, and pretraining could be the actual source of retention.

The practical payoff is a retrofit test. We add a merged large-kernel branch to a pretrained small-kernel network, then compare its forgetting to the untouched network and to the native large-kernel model.

Baselines

The naive reference point is the 7×7 depthwise backbone trained by plain sequential fine-tuning with no continual-learning mechanism. Every other arm is judged by how much of this baseline's forgetting it removes.

The strongest competitor for the scientific question is the same-backbone global-attention arm, which reproduces the Transformer side of Rethinking Architecture Design… under matched conditions. The comparison is fair because both arms share:

  • data order;
  • epochs per task;
  • optimizer;
  • a per-arm learning-rate and weight-decay sweep of the same size (eight configurations), selected on a held-out validation split of the first task only.

That paper reports, on CIFAR-10 Split-3 under parallel federated averaging, a 37.34% accuracy boost for ViT over ResNet. It also reports ViT at about 80% vs. ResNet at about 50% on the Retina dataset in the extreme one-sample-per-client setting. These are the gaps we aim to explain. The large-kernel convolution "beats" this baseline for our purposes if its final average accuracy under sequential transfer is within one standard deviation (five seeds and five task orders) of global attention, while its measured forgetting is at least half-way from the 7×7 arm toward the attention arm.

The strongest existing continual method on class-incremental CIFAR-100 in the file is ICON, which reports 87.07% accuracy vs. 86.13% for CLAP (Understanding Catastrophic Interference…). In the online, task-free setting it is CVT, which reports 24.45% on 10-split CIFAR-100 with a 500-sample buffer vs. 16.28% for the previous best, ERT (Online Continual Learning with Contrastive Vision Transformer).

We do not expect a bare architecture to beat these methods. Instead we apply experience replay with the same 500-sample buffer on every arm and ask whether the large-kernel backbone narrows the gap to CVT's reported figure at equal parameter count. CVT uses about 8.9M parameters, so we build 9M-parameter variants for this comparison.

For parameter-isolation methods we use APD, which reports about 60.7% vs. 53.7% for EWC on CIFAR-100 splits and an order disparity of 1.79% vs. 7–8% for baselines (Scalable and Order-robust Continual Learning…). We run APD on the 7×7 and 31×31 backbones to test whether reach and isolation add up.

The key ablations are the windowed-attention arm (routing without reach), the dilated arm (nominal reach without dense coverage), and the BatchNorm/LayerNorm crossing.

Why not just…?

Why not just use a Vision Transformer, since it already has a global receptive field? If the attention arm is the only one that retains knowledge, that is the answer and we will report it. The file supplies two reasons it is not obviously enough. Attention costs grow quadratically with image resolution, which matters for dense prediction (HyenaPixel). And plain ViTs still drift under streaming updates without added machinery (Continual Learning with Lifelong Vision Transformer). The factorial decides whether cheaper convolutional reach gets the same benefit.

Why not just swap BatchNorm for LayerNorm in a ResNet? This is a genuine threat. BatchNorm's running statistics are tied to each client's data. The CNN-versus-Transformer gap in the federated study coincides with a BatchNorm-versus-LayerNorm gap, and that paper does not separate the two. We cross normalization with the mixer explicitly. If the LayerNorm 7×7 arm closes most of the gap, our reach hypothesis is largely refuted, and we will say so.

Why not just add experience replay to a small-kernel CNN? Replay is applied equally to all arms, so we will directly see whether it makes backbone choice irrelevant. Replay can provably increase forgetting proves that small buffers can turn vanishing forgetting into constant-order forgetting when task subspaces are unfavorably aligned. Our gradient-alignment measurements test whether large receptive fields change that alignment and thereby change when replay helps.

Why not just widen the receptive field cheaply with dilated convolutions? The dilated arm tests exactly this. We predict it will match the nominal reach of the 31×31 kernel but not its measured effective receptive field, and will therefore forget more. If instead it matches the dense large kernel, the cheap option wins and the paper's recommendation changes accordingly.

Proposed Experiments

The first experiment is sequential cyclic weight transfer on the CIFAR-10 Split-3 and Retina partitions of Rethinking Architecture Design…. This is domain-style shift and the setting where the original CNN collapse was seen.

The second is class-incremental 10-split CIFAR-100 and ImageNet-100, both offline and online with a 500-sample buffer. We report:

  • final average accuracy;
  • average forgetting, meaning the drop from each task's best accuracy;
  • backward transfer;
  • the order-disparity metric from APD, computed over five task orders.

The third is continual semantic segmentation on Pascal VOC 19-1 and ADE20K with LGKD's output-level distillation (Label-Guided Knowledge Distillation…) held fixed while the backbone varies. This asks whether wider context reduces confusion between novel classes and background. LGKD reports a +76% relative gain in novel-class mean intersection-over-union over MiB on VOC 19-1. No absolute ADE20K continual numbers were found on Lacuna for LGKD, so we will establish our own reference.

The decisive check is the factorial pattern on the cyclic-transfer benchmark. Reach is supported if the 31×31, SLaK, WTConv, and HyenaPixel arms cluster with global attention, while windowed attention clusters with 7×7. Routing is supported if the pattern is reversed. We also expect forgetting to correlate with measured effective receptive field across all convolutional arms, with the correlation weakening on class-incremental CIFAR-100. The retrofit experiment turns a positive result into a usable recipe: forgetting in the retrofitted small-kernel network should fall toward the native large-kernel level.