Convolutional networks apply the same filter at every image position, so their features shift when the input shifts. Classifiers then average those features over the whole image, which makes the final prediction nearly insensitive to where an object sits. This is usually treated as a virtue. Recent theory says it has a cost: a strictly shift-invariant classifier can draw its decision boundary using only a small part of the input signal, so its safety margin against small malicious pixel changes shrinks as images get larger. A separate line of work defends against such changes by training on worst-case perturbed images, and that defense carries its own cost: accuracy on ordinary, unperturbed images drops. This note asks whether part of that accuracy cost comes from the architecture rather than the training objective. If so, giving the network controlled access to absolute position should move the trade-off.
The proposal is simple to state. Append coordinate channels to a convolutional network, and add random noise of scale $\sigma$ to those coordinates during training. This gives a single dial. At $\sigma = 0$ the network knows exactly where every pixel is. As $\sigma$ grows, it knows less and less, and at large $\sigma$ it falls back to ordinary translation invariance. We train adversarially across the whole range of $\sigma$ and map how clean accuracy, robustness to pixel-level attacks, and robustness to global shifts and rotations change along it. The claim to test is that this one architectural knob traces a better three-way trade-off curve than reweighting the training loss, which is the expensive route taken by Pareto adversarial training, and that it does so at the cost of ordinary adversarial training.
Core Claim and Why It Is Open
The motivating evidence comes from Shift Invariance Can Reduce Adversarial Robustness. That paper proves that a shift-invariant linear classifier's margin can depend only on the spatial mean of the input, so the margin shrinks as $O(1/\sqrt{d})$ for $d$ pixels. It extends the result to infinitely wide CNNs. Empirically, it shows that networks with higher shift consistency, fully connected networks, AlexNet without global pooling, and Vision Transformers line up as expected: the less shift-invariant models are more robust. The paper does not test the most direct intervention, which is to break invariance by handing the network coordinates. It also does not measure what happens to clean accuracy under adversarial training.
The method we reuse comes from two papers. CoordConv (An intriguing failing of convolutional neural networks and the CoordConv solution) concatenates normalized $x, y$ (and optionally radius) channels to feature maps. It is notable for changing ImageNet accuracy by only +0.04%. That result says coordinates are harmless for clean classification, but it says nothing about robustness. GeoPos (GeoPos: A Minimal Positional Encoding for Enhanced Fine-Grained Details in Image Synthesis Using Convolutional Neural Networks) adds a random shift to the coordinate channel during training. Its purpose there was to stop generators from memorizing absolute canvas positions. We repurpose that random shift as a continuous control variable and study it under adversarial attack, which GeoPos never did.
The main constraint comes from Pareto Adversarial Robustness. It found that robustness to pixel-norm attacks is negatively correlated with robustness to rotation-translation attacks, and it resolves the conflict with multi-objective optimization over loss weights. This finding cuts against any naive "coordinates make everything better" story. A network that knows absolute position should be more exposed to an attacker who simply translates the image. We therefore expect a real trade-off along $\sigma$, not a free lunch. The question is whether the curve traced by the architectural dial lies outside the curve traced by loss reweighting.
Concretely, with coordinate channel $c(u) + \epsilon$ at pixel $u$, where $\epsilon \sim \mathcal{N}(0, \sigma^2)$ is drawn once per image and shared across pixels and channels, we train
$$\min_\theta \; \mathbb{E}_{(x,y),\,\epsilon}\left[\max_{\|\delta\|_\infty \le 8/255} \mathcal{L}\big(f_\theta(x+\delta,\; c+\epsilon),\, y\big)\right].$$
The attacker controls only the pixels $\delta$; the coordinate channel is part of the network. At test time we set $\epsilon = 0$, so the evaluated model is deterministic. This matters, and we return to it below.
Research Program
The work has three stages. The first is a cheap mechanism check on the synthetic dot-on-gray-background task from the shift-invariance paper. On that task, a shift-invariant CNN fails at a perturbation of about $1/\sqrt{d}$, while a fully connected network stays near 1.0 regardless of image size. We will measure the minimum fooling perturbation as a function of $d$ for a CNN with global pooling plus coordinate channels at several values of $\sigma$. If the margin argument transfers, the $\sigma = 0$ curve should flatten toward the fully connected curve, and increasing $\sigma$ should bend it back toward $1/\sqrt{d}$. If adding coordinates leaves the $1/\sqrt{d}$ scaling intact, the network is ignoring them, much as the +0.04% ImageNet result suggests it might. The theory would then not apply to this intervention, and we would know before spending GPU time on CIFAR.
The second stage is the main study on CIFAR-10, with CIFAR-100 as a check. We use ResNet-18 and the standard 10-step projected-gradient-descent adversarial training at $\ell_\infty$, $\epsilon = 8/255$. We sweep $\sigma$ over roughly six values, from zero to beyond the coordinate range, and vary whether coordinates enter at the input only, at every stage, or only before pooling. For each model we record five quantities. The first is clean accuracy. The second is AutoAttack accuracy, where AutoAttack is a standard ensemble of strong gradient-based and black-box attacks. The third is worst-case accuracy over a grid of rotations and translations, following the threat model of the Pareto paper. The fourth is pixel-flow attack accuracy, and the fifth is shift consistency, meaning how often the prediction is unchanged under small translations. Shift consistency is the proposed mediator: the shift-invariance theory predicts robustness should track it, so we test whether $\sigma$ moves robustness through shift consistency or through some other route. We also record training-time cost, since the practical case for an architectural dial rests on its being cheaper than Pareto training.
The third stage is conditional on stage two showing an effect. We would move to a task where absolute position is genuinely required: SOLO-style instance segmentation on MS COCO. SOLO assigns object identity by grid location and feeds CoordConv channels into its mask branch (SOLO: Segmenting Objects by Locations, SOLOv2: Dynamic and Fast Instance Segmentation). This is the "coloring dilemma" regime identified by Semi-convolutional Operators for Instance Segmentation, where a translation-invariant network cannot separate identical neighboring objects. There we would ask whether small pixel attacks can merge or swap instance masks, and whether the jitter dial trades mask AP against that vulnerability the way it trades accuracy on CIFAR. None of SOLO, SOLOv2, or Benchmarking the Robustness of Instance Segmentation Models tested coordinate-dependent heads under attack. The benchmark paper evaluated common image corruptions only.
Baselines
The naive reference point is a ResNet-18 with global average pooling, trained with 10-step projected-gradient-descent adversarial training under identical settings and no coordinates. The closest reported figure in the file is from Understanding and Mitigating the Tradeoff between Robustness and Accuracy: 87.3% standard accuracy and 45.8% robust accuracy on CIFAR-10 at $\epsilon = 8/255$. We will reproduce our own number on our architecture and budget rather than compare against that figure directly. Beating this baseline means raising clean accuracy, AutoAttack accuracy, or both, at some $\sigma$ without losing rotation-translation accuracy.
The strongest competitor on the specific question of the three-way trade-off is Pareto adversarial training from Pareto Adversarial Robustness. It solves a quadratic program to balance clean, pixel-norm, flow, and rotation-translation losses, and it reports a CIFAR-10 universal robustness score of 130.96, against 127.11 for averaged adversarial training and 109.98 for worst-case adversarial training. We will run it on the same backbone and data, with the same number of training epochs, and give it a hyperparameter search budget equal to our $\sigma$ sweep. Our method beats it if some point on the $\sigma$ curve dominates some point on its front in all three robustness measures plus clean accuracy, or matches its universal score at plain adversarial-training cost. A finer test is whether combining the two (Pareto training plus jitter) pushes its front outward.
The strongest existing architectural defense is deformable convolution with stochasticity from Adversarial Robustness via Deformable Convolution with Stochasticity. It reports 75.46% AutoAttack accuracy on CIFAR-10 and a 7.94% clean-accuracy gain on ResNet-18. Because that defense is randomized at inference, we will re-evaluate it with expectation over transformation, which averages gradients over the model's own randomness. Without this, randomness can make gradient attacks look weaker than they are. Our deterministic test-time setting avoids this issue by construction, and we will report both. We do not expect to beat its headline number. The relevant comparison is with expectation-over-transformation attacks applied to both methods.
Robust self-training from the same trade-off paper reports 91.8% clean and 58.5% robust accuracy, but it uses 500,000 extra unlabeled images. We treat it as an orthogonal addition, not a head-to-head rival, and run jitter with and without it.
Two ablations isolate the key ingredient. The first removes global average pooling and uses a flattened head, with no coordinates. The second keeps pooling, adds coordinates, and sets $\sigma$ very large, which matches parameter count while destroying positional information. If the first ablation matches the $\sigma = 0$ model, coordinates add nothing beyond removing pooling. If the second matches the jitter model at intermediate $\sigma$, the benefit comes from extra input channels or noise, not from position.
Why not just…?
Why not just train with random translations and crops? Augmentation pushes the network toward higher shift consistency, which is exactly the property the shift-invariance paper links to smaller margins. Augmentation is still the natural way to gain rotation-translation robustness, so it is a real competitor on that axis. We will run translation augmentation at several strengths as its own curve. If that curve matches the $\sigma$ curve, the dial is just a costlier augmentation, and we will say so. CoordConv's held-out-quadrant failure for standard CNNs (86% after an hour with 200,000 parameters, versus 100% in 15 seconds for CoordConv) suggests augmentation cannot supply absolute position, but that evidence comes from a different task.
Why not just remove global pooling or use a Vision Transformer? This is the most serious threat. The shift-invariance paper already finds AlexNet-style and transformer models more robust, so the robustness gain might come entirely from dropping pooling. The flattened-head ablation answers this directly. We will also include a small Vision Transformer trained adversarially, and apply the same jitter to its positional embeddings. If jitter traces a similar curve there, that strengthens the claim that the dial concerns positional information itself, not convolution.
Why not just add unlabeled data with robust self-training? Robust self-training addresses the trade-off as a shortage of training data. It needs a large in-distribution unlabeled pool and does not address the conflict between pixel-norm and rotation-translation robustness. The stacked experiment shows whether the two gains add. If jitter's gain disappears once robust self-training is applied, the architectural effect was only a small-data effect, which the trade-off paper's theory would predict.
Decisive Checks
Three results would settle the proposal. On the synthetic task, the coordinate network's fooling distance should stop shrinking as $1/\sqrt{d}$ at $\sigma = 0$ and should return to that scaling as $\sigma$ grows. On CIFAR-10, the $\sigma = 0$ model should beat the flattened-head ablation on AutoAttack, and some intermediate $\sigma$ should sit outside Pareto adversarial training's front at a fraction of its training cost. Shift consistency should move monotonically with $\sigma$ and predict robustness across the sweep. If the network learns to ignore coordinates, or if removing pooling explains everything, the margin-based explanation for this architectural intervention is wrong, and that negative result is itself worth reporting given that the theory predicts otherwise.
References
- Shift Invariance Can Reduce Adversarial Robustness
- An intriguing failing of convolutional neural networks and the CoordConv solution
- GeoPos: A Minimal Positional Encoding for Enhanced Fine-Grained Details in Image Synthesis Using Convolutional Neural Networks
- Pareto Adversarial Robustness: Balancing Spatial Robustness and Sensitivity-based Robustness
- SOLO: Segmenting Objects by Locations
- SOLOv2: Dynamic and Fast Instance Segmentation
- Semi-convolutional Operators for Instance Segmentation
- Benchmarking the Robustness of Instance Segmentation Models
- Understanding and Mitigating the Tradeoff between Robustness and Accuracy
- Adversarial Robustness via Deformable Convolution with Stochasticity