Making an image classifier robust to small adversarial perturbations usually costs it several points of accuracy on ordinary, unperturbed images. The usual explanation is that robust training forces the network to give up brittle but predictive features. A second line of evidence points somewhere more mechanical. Wider networks become more sensitive to input perturbations, and certain block shapes, namely blocks that expand channels internally, are measurably worse for robustness. This note proposes that these two architectural observations are one phenomenon. What makes a network fragile is the width of the hidden layer inside each residual branch, not the width of the network as a whole. The width of the main residual pathway, by contrast, is what buys clean accuracy.
If that is right, the classic bottleneck block is the natural knob for separating the two. This is the block that squeezes channels with a cheap 1×1 convolution, does its spatial filtering in that narrow space, and expands back, as in Going deeper with convolutions. A network with a wide residual pathway and narrow internal bottlenecks should sit on a better robustness–accuracy curve than either a uniformly wide network or an inverted-bottleneck network with the same parameter budget. It should also get there without the heavy regularization that otherwise taxes clean accuracy. The work is a controlled architecture study with a mechanistic hypothesis and a falsifiable prediction. It is not a new defense module.
Core Claim
Does Network Width Really Help Adversarial Robustness? splits robust accuracy into two parts. The first is correct classification. The second is perturbation stability, meaning the prediction stays fixed within the allowed perturbation ball. The paper's analysis uses the neural tangent kernel, a linearized description of very wide networks, and shows that the local Lipschitz constant (how much the output can change per unit of input change) grows like $\sqrt{m}$ with width $m$. Its remedy is to scale the robustness regularizer up with width (Width Adjusted Regularization, WAR). Separately, RobArch trained over 150 models at a fixed parameter budget and found two things: inverted bottlenecks hurt robustness, and deeper-slimmer networks beat wider-shallower ones. Neither paper isolates the ratio between a block's internal width and its pathway width. The WAR analysis treats width as uniform. RobArch reports the inverted-bottleneck effect without explaining it.
Our hypothesis is that the $m$ that matters is $C_{\text{mid}}$, the internal width of each residual branch. For a block $x \mapsto x + g(x)$ with $g = W_{\text{up}} \circ \sigma \circ K \circ \sigma \circ W_{\text{down}}$, where $W_{\text{down}}, W_{\text{up}}$ are the 1×1 projections, $K$ is the 3×3 spatial convolution and $\sigma$ is the activation, the block's sensitivity is bounded by $1 + \|W_{\text{up}}\|\,\|K\|\,\|W_{\text{down}}\|$. All three factors live in, or map through, the $C_{\text{mid}}$-dimensional space. An inverted bottleneck makes $C_{\text{mid}}$ larger than the pathway width $C$, which on this view is exactly what RobArch observed hurting robustness. A classic bottleneck makes $C_{\text{mid}}$ small while leaving the pathway wide.
The prediction follows directly. Perturbation stability should track $C_{\text{mid}}$, and clean accuracy should track pathway width and total capacity. Shifting capacity from the internal width to the pathway should therefore move the whole trade-off curve outward, and not merely slide along it. Extending the $\sqrt{m}$ analysis to the hidden width of a residual branch is our conjecture, not a result of the cited paper. Testing it is the point of the study.
This also gives a structural reading of a debate in the literature. Robustness May Be at Odds with Accuracy argues that the clean-accuracy cost is intrinsic. A Closer Look at Accuracy vs. Robustness shows that CIFAR-10's classes are separated by 3–7× the attack radius. It concludes that much of the cost comes from a large robust generalization gap: a 16.33% train–test clean-accuracy gap under standard adversarial training on CIFAR-10. That paper also argues that closing the gap needs changes to network topology, not just regularization. Our study tests one concrete topological lever of the kind it calls for.
Research Program
We first build a family of residual networks for CIFAR-10 and CIFAR-100 in which two quantities are set independently: the pathway width $C$, and the internal ratio $r = C_{\text{mid}}/C$. The ratio spans inverted blocks ($r = 4$, MobileNetV2-like), plain blocks ($r = 1$), and classic bottlenecks ($r = 1/2, 1/4, 1/8$, the last approaching the aggressive squeeze of SqueezeNext). Two series control for confounds. In the first, parameters and FLOPs are matched by adjusting depth. In the second, depth is matched and parameters are allowed to vary. Following Conv-Adapter's observation (Conv-Adapter) that pointwise projections alone lose spatial context, every variant keeps a 3×3 spatial convolution inside the bottleneck, so spatial capacity is not a hidden variable. RobArch's other recommendations (squeeze-excitation, SiLU activations, stem changes) are held fixed across variants so that only $r$ and $C$ move.
Every architecture is adversarially trained with TRADES (Theoretically Principled Trade-off between Robustness and Accuracy). Its coefficient $\beta$ explicitly trades clean error against boundary error, so sweeping $\beta$ traces a full trade-off curve for each architecture rather than a single point. This follows the trade-off-curve methodology of Architecture Selection via the Trade-off Between Accuracy and Robustness, which found that architecture family matters more than raw capacity. We take that finding down to a single within-block dimension.
Alongside accuracy, we measure the mechanism directly. First, we use Wu et al.'s split of robust accuracy into correct classification and perturbation stability. Second, we estimate the empirical Lipschitz constant of each residual branch with power iteration on the input–output Jacobian, a cheap way to find a matrix's largest stretching factor. Third, we measure how noise-sensitive the channels are at each stage. For this we reuse the noise-injection procedure of Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck. That method separates channels that tolerate injected noise from those that do not, and the paper applied it mainly at the last layer. We apply it inside each bottleneck. This lets us ask whether narrow internal widths simply carry fewer noise-sensitive channels, or whether they reorganize what those channels carry.
If the CIFAR results support the claim, we scale to ImageNet. There the comparison is ResNet-50 against a variant with a wider pathway and narrower bottleneck at matched parameters, both trained under the same fast adversarial training recipe.
Baselines
The naive reference point is a standard ResNet-18 trained with standard PGD adversarial training (PGD is the iterative gradient attack used to generate training perturbations). Improving Adversarial Robustness via Information Bottleneck Distillation reports 47.73% AutoAttack accuracy for this setup on CIFAR-10. We retrain it under our exact recipe rather than borrowing the number.
The strongest width-specific competitor is WAR on WideResNet-34-10. It attacks the same $\sqrt{m}$ instability we target, but by scaling regularization rather than reshaping blocks, and Wu et al. report 54.73% AutoAttack accuracy on CIFAR-10. No reported clean accuracy for WAR was found on Lacuna, so we will reproduce it. We give WAR the same epoch budget and the same $\beta$-grid size that each of our architectures receives.
The strongest architecture-search competitor is the RobArch-S design rules applied at CIFAR scale. They encode "deeper and slimmer" without separating internal from pathway width, so any gain we report must exceed what depth alone delivers at matched parameters. For clean-accuracy-oriented training methods, TART (TART) reports 83.73% clean and 36.58% AutoAttack accuracy against 81.76% and 35.89% for standard training with WideResNet-32-10. It is orthogonal to architecture and serves as a reference for how large a clean-accuracy gain counts as meaningful at fixed robustness.
The key ablation is $r$ itself: the same pathway width and depth with $r$ moved from 4 to 1/8. A further spectral-normalized variant at $r = 1$ tests whether explicitly capping branch sensitivity reproduces the effect.
Beating a baseline means one thing here: higher clean accuracy at matched AutoAttack accuracy, read off the interpolated trade-off curves, with matched parameters and FLOPs. A gain of at least two clean points, about the size of TART's improvement, at equal robustness over both WAR and the depth-matched RobArch-style network would count as a real outward shift of the trade-off curve.
Why not just…?
Why not just tune the TRADES or WAR coefficient harder? Turning up regularization moves a model along its trade-off curve. Our claim is that architecture moves the curve itself. The $\beta$ sweep settles this directly. If every bottleneck variant's curve coincides with the uniform-width curve once $\beta$ is retuned, the hypothesis is false and the effect was only implicit regularization. WAR is a serious threat because it already removes much of the width-driven instability. The decisive comparison is clean accuracy at equal robustness, not robustness alone.
Why not just go deeper and slimmer as RobArch recommends? This is the closest threat, because narrowing blocks overall also narrows $C_{\text{mid}}$. Our matched-depth series separates the two explanations. If a wide-pathway, narrow-bottleneck network beats a uniformly slim network of equal depth and parameter count on clean accuracy at equal robustness, then the internal ratio carries information that "slimness" does not. If the two match, we will report that RobArch's rule already captures the effect.
Why not just constrain each layer's Lipschitz constant directly, for example with spectral normalization? That controls the same bound in our core equation, but it does so by shrinking every layer's weights. The file reports that Lipschitz-style constraints typically cost 5–15% clean accuracy on CIFAR-10 and ImageNet. Narrowing $C_{\text{mid}}$ limits how many directions a block can amplify rather than how much it amplifies along each, so pathway capacity should be left intact. The spectral-norm ablation at $r = 1$ shows which route costs less clean accuracy for the same measured branch sensitivity.
Why not just add dropout or unlabeled data? Yang et al. find that dropout helps but does not close the gap. Their robust self-training with dropout on SVHN reaches 95.19% clean and 55.22% adversarial accuracy and still leaves a gap. These methods are complementary to architecture rather than competing with it, so we will apply dropout uniformly across all variants and check that the ranking of $r$ values survives.
Decisive Checks
All robust numbers come from the full AutoAttack ensemble under RobustBench's protocol (RobustBench). That paper documents that original publications often overstated robust accuracy by more than 10%. This matters here because narrow layers could in principle mask gradients rather than confer real robustness. We therefore also report the gradient-free Square Attack component separately, and flag any variant whose white-box and black-box numbers diverge.
The central experiment fixes the pathway width and varies $r$, then fixes $r$ and varies the pathway width. The hypothesis survives only if two things hold together. Perturbation stability and branch Lipschitz estimates must move with $C_{\text{mid}}$ and stay roughly flat in pathway width. Clean accuracy must show the reverse pattern.
A second check targets RobArch's inverted-bottleneck finding. We take an inverted-bottleneck network, insert a narrow 1×1 squeeze into each branch, and leave everything else unchanged. If the robustness deficit disappears, internal width is the cause.
Finally, we compare the stage-wise channel noise sensitivity of the best and worst variants. If narrow bottlenecks retain clean accuracy while carrying fewer noise-sensitive channels, the brittle-feature story and the width story become one account, measured inside the network rather than only at its output.
References
- Going deeper with convolutions
- Do Wider Neural Networks Really Help Adversarial Robustness?
- RobArch: Designing Robust Architectures against Adversarial Attacks
- Robustness May Be at Odds with Accuracy
- A Closer Look at Accuracy vs. Robustness
- SqueezeNext: Hardware-Aware Neural Network Design
- Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets
- Theoretically Principled Trade-off between Robustness and Accuracy
- Architecture Selection via the Trade-off Between Accuracy and Robustness
- Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck
- Improving Adversarial Robustness via Information Bottleneck Distillation
- TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training
- RobustBench: a standardized adversarial robustness benchmark