Image classifiers often change their answer when the picture moves by one pixel. Current fixes guarantee stability only for "cyclic" shifts, where pixels that leave one edge of the image wrap around to the other. Real cameras do not wrap: when the view moves, new pixels enter the frame and old ones leave. This note proposes to attack the remaining failure from a different direction, taken from the old literature on distributed memories. A memory that spreads each item across many units turns random damage into mild background noise. The question here is whether a network that spreads its evidence for each class across many feature channels, instead of a few dominant ones, absorbs the small, roughly random feature changes that a realistic shift causes in the same way.
The claim can be tested and might be false. A shift changes the logit, the network's pre-softmax score for a class, by a weighted sum of per-channel feature changes. If those changes are only weakly correlated across channels, spreading the readout weights shrinks that sum relative to the class margin, much as averaging shrinks noise. If the changes are strongly correlated, because every channel inherits the same aliased low-level edge, spreading does nothing. Aliasing here means the distortion produced when a downsampling layer samples features too coarsely. We will measure which case holds. We will then test whether a cheap "evidence spreading" penalty, together with existing channel-decorrelation layers, lowers the flip rate under realistic shifts beyond what anti-aliasing and polyphase methods achieve. A side benefit can be checked directly: the same property should also make the network tolerant to dead channels and bit flips.
Core Claim
Write the change in the logit margin under a one-pixel realistic shift as $\Delta = \sum_i w_i \delta_i$. Here $w_i$ is the final-layer weight difference between the predicted class and the runner-up for pooled channel $i$, and $\delta_i$ is the shift-induced change in that channel's pooled activation. With $\Sigma$ the covariance of the vector $\delta$ over shifts and images, the spread of $\Delta$ is $\sqrt{w^\top \Sigma w}$, and a prediction flips roughly when
$$ \frac{m}{\sqrt{w^\top \Sigma w}} \lesssim 1, $$
where $m$ is the clean logit margin. When $\Sigma$ is close to diagonal, spreading a fixed margin over $k$ equally weighted channels reduces the denominator by about $\sqrt{k}$. That figure is the standard averaging argument. It is the same logic by which Invariant Object Recognition Using a Distributed Associative Memory showed that a matrix memory improves its output-to-input noise ratio and survives the destruction of 75–90% of its elements. When $\Sigma$ has one dominant direction aligned with $w$, spreading gives no benefit.
The proposal's central claim is that trained convolutional networks sit closer to the first regime than to the second once the cyclic part of the aliasing is removed. On that view, distributed readout is a missing complement to architectural shift fixes rather than a replacement for them.
Relation to Prior Work
The closest bridge between the two directions is Laplace-HDC. That work builds binary hyperdimensional classifiers, which represent inputs as very long random binary vectors. It gives them translation awareness through 2D cyclic permutations of the vector coordinates and shows they keep over 80% accuracy with 25% of bits flipped. It engineers exact equivariance for cyclic shifts on MNIST-scale data with hand-made Haar filters. We do the opposite: we keep a standard learned deep network and ask whether distributedness itself, not an engineered permutation, buys stability under non-cyclic shifts. The Wechsler et al. system makes translation invariance exact through a Fourier-magnitude pipeline and needs centered objects. We borrow only its noise-averaging argument.
The method we reuse comes from Channel Equilibrium Networks. That paper adds decorrelation-and-reweighting blocks that prevent channels from collapsing toward zero. It shows that the resulting networks degrade gracefully under channel ablation and gain +1.6% top-1 on ResNet-50 for a 0.48% increase in compute. It never measures shift behavior, and that is the gap we test.
The motivating evidence comes from the shift literature. Why do deep convolutional networks generalize so poorly to small image transformations? reports top-1 changes up to 30% of the time under one-pixel shifts. Lost in Translation shows that DINOv2's nearest-neighbor representation changes 40% of the time under one-pixel realistic shifts, and that scale does not cure this.
The constraints also come from that literature. Truly Shift-Invariant Convolutional Neural Networks introduces adaptive polyphase sampling (APS), which keeps whichever subsampling grid has the largest norm. APS guarantees full consistency, but only under circular shifts with circular padding, and it shows that nonlinearities spoil pure low-pass fixes. Making Convolutional Networks Shift-Invariant Again introduces BlurPool, a fixed blur before subsampling, which gains only 1.3–2.1% in consistency. On Translation Invariance in CNNs and Mind the Pad show that border padding gives networks absolute position cues. Those cues are exactly the part of realistic shifts that cyclic guarantees ignore, and they are a likely source of correlated $\delta$ that spreading cannot fix.
Shift Invariance Can Reduce Adversarial Robustness warns that enforcing invariance can cost robustness to adversarial attacks. We therefore report adversarial accuracy alongside every intervention.
Research Program
The first stage is diagnostic and needs no new training. For pretrained ResNet-18/50 models, with and without BlurPool and APS, we compute $\delta$ at the global-pooling layer under one-pixel realistic shifts. We generate these following the crop protocol of Shifman and Weiss, taking a larger image and cropping a window that moves by one pixel. We then estimate $\Sigma$, its effective rank, and how strongly its top eigenvectors align with each class's readout direction $w$.
We also measure how concentrated each prediction's evidence is. For this we use the participation ratio of the per-channel contributions $w_i f_i$, a number that counts roughly how many channels carry the evidence. We check whether flips are predicted by the ratio in the core equation better than by margin alone.
A second diagnostic splits $\delta$ into two parts. One is the cyclic-shift component, which APS removes by construction. The other is the boundary component, from pixels entering and leaving the frame. This split tells us which part is decorrelated across channels and so open to averaging.
The second stage is an intervention. We train networks with an evidence-spreading penalty that adds to the loss the negative log participation ratio of the true-class contributions $w_i f_i$, weighted by a coefficient tuned on held-out data. We train these both alone and on top of Channel Equilibrium blocks and APS. We report whether realistic-shift flip rate falls, whether clean accuracy holds, and whether channel-ablation and bit-flip curves improve at the same time. That joint improvement is what would make "distributed" a single robustness property rather than a coincidence.
Baselines
The naive reference is a standard ResNet-18 on CIFAR-10 and ResNet-50 on ImageNet with zero padding, trained with ordinary random-crop augmentation. Chaman and Dokmanic report 91.96% CIFAR-10 accuracy for the ResNet-18 version. On ImageNet-P, from Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, a standard ResNet-18 has a 15.6% flip probability between consecutive frames.
The strongest architectural competitor is APS. It reaches 94.53% CIFAR-10 accuracy with 100% circular consistency, against 94.01% for BlurPool. It is the strongest competitor because it already removes the cyclic part of the problem exactly. For APS under realistic one-pixel shifts on CIFAR-10, no reported result was found on Lacuna, so we will measure it ourselves.
The strongest test-time competitor is Robust Inference by Crop Selection (RICS) from Lost in Translation. It picks a sub-crop by a deterministic local-maximum score and cuts DINOv2's representation-change rate to under 5% at roughly a 1% top-1 cost.
All trained baselines share our data, augmentation, schedule, and hyperparameter search budget, with the same number of tuning runs per method. RICS is applied to the same trained backbones.
Beating the field means two things on realistic one-pixel shifts. First, APS plus spreading must lower the flip rate relative to APS alone by more than the seed-to-seed spread across five seeds, at no more than 0.5 points of clean accuracy cost. Second, any remaining gap to RICS must be achieved without its test-time crop search.
The key ablation removes the spreading penalty but keeps Channel Equilibrium blocks. A second ablation replaces the penalty with dropout at matched clean accuracy. Together they isolate whether explicitly spreading readout evidence, rather than generic regularization, is what matters.
Why not just…?
Why not just train with shift augmentation and a consistency loss? This is the most serious threat. Directly penalizing the change in output between an image and its shifted copy targets the ratio in the core equation head-on, and it may beat spreading on shift metrics alone. We include it as a fourth baseline at matched compute. If it dominates on flip rate, the proposal's surviving contribution is narrower: whether spreading also gives fault tolerance under channel ablation and bit flips, which consistency training has no reason to provide. We would also report that combining the two is the useful outcome. We will state this result plainly rather than argue around it.
Why not just use APS or BlurPool, since they address the cause? They address the cyclic cause. The Shifman–Weiss results show that realistic shifts survive these fixes, and the padding studies identify border cues as a residual source. Our cyclic-versus-boundary split of $\delta$ shows directly whether the residual left by APS is the decorrelated kind that spreading can absorb. If it is not, the idea fails at the diagnostic stage at low cost.
Why not just use dropout, which already forces distributed features? Dropout spreads features but does not specifically spread the evidence feeding the decision margin, and it can lower the margin itself. The matched-accuracy dropout ablation settles whether the explicit penalty adds anything.
Decisive Checks
The idea lives or dies on three measurements, all taken on CIFAR-10 and ImageNet with the realistic-shift crop protocol and with ImageNet-P flip probability as a secondary metric. The first is whether the residual shift perturbation left after APS has high effective rank and low alignment with readout directions. The second is whether evidence concentration predicts which images flip, after controlling for margin. The third is whether the spreading penalty moves the realistic flip rate and the channel-ablation curve together, in the same direction, without the adversarial-robustness loss that the invariance–robustness trade-off paper warns about.
A low-rank, readout-aligned residual would falsify the averaging account. Such a result would itself be informative, because it would point to specific shared low-level channels as the place where realistic shift sensitivity enters.
References
- Invariant Object Recognition Using a Distributed Associative Memory
- Laplace-HDC: Understanding the Geometry of Binary Hyperdimensional Computing
- Channel Equilibrium Networks for Learning Deep Representation
- Why do deep convolutional networks generalize so poorly to small image transformations?
- Lost in Translation: Modern Neural Networks Still Struggle with Small Realistic Image Transformations
- Truly Shift-Invariant Convolutional Neural Networks
- Making Convolutional Networks Shift-Invariant Again
- On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location
- Mind the Pad - CNNs Can Develop Blind Spots
- Shift Invariance Can Reduce Adversarial Robustness
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations