Plug-in attention modules for convolutional networks make two claims. CBAM, Coordinate Attention and SCSA each say that learning which channels matter (channel attention) and where in the image to look (spatial attention) adds one to two points of ImageNet accuracy to a ResNet or MobileNet. Almost every one of these gains was measured against a backbone trained with an old, weak recipe. SCSA's ResNet-50 baseline sits near 76.4%. A vanilla ResNet-50 trained with a modern recipe reaches 80.4% with no architectural change at all. We do not know how much of the reported benefit of attention is real and how much is compensation for missing regularization.
We propose a controlled factorial study to answer a sharper question than "does attention still help." We test a specific substitution hypothesis. Modern recipes include augmentations that delete rectangular image regions (CutMix, Random Erasing), and these already teach a network to find signal wherever it survives. If that is right, the spatial half of dual attention should lose most of its value under such augmentations, while the channel half keeps its value. We test this by switching the region-deleting augmentations on and off under an equal tuning budget. We then look directly at whether learned spatial masks collapse toward constants, and at whether ImageNet rankings of attention modules predict their rankings on segmentation and detection.
Core Claim
The claim is falsifiable. Take the gain from input-dependent spatial gating and measure it under a recipe with CutMix and Random Erasing. We predict it falls to less than half of its value under the same recipe with those two augmentations removed. Channel gating (the Squeeze-and-Excitation style: pool each channel to one number, then reweight channels) should retain most of its gain. A second claim is that once the backbone is trained well, any remaining advantage of CBAM, Coordinate Attention or SCSA over plain SE is within seed noise on ImageNet. Any real differences among them would then appear, if at all, on dense prediction.
What is new is the interaction measurement, meaning attention type crossed with augmentation type, together with a mechanistic readout of the masks themselves. A re-benchmark alone would not be new: it would only replace one leaderboard with another.
Relation to Prior Work
Three papers supply the modules under test. CBAM applies channel attention and then spatial attention through a 7×7 convolution. It reports ResNet-50 top-1 error dropping from 24.56% to 22.66%, against 23.14% for SE. Coordinate Attention replaces 2D pooling with horizontal and vertical 1D pooling, lifting MobileNetV2 from 72.3% to 74.4% (SE: 73.5%). SCSA puts a multi-scale spatial prior before channel self-attention, reports 77.49% on ResNet-50 and +0.94 mIoU on ADE20K, and concedes throughput overhead.
All three compare modules under the default recipe of the base network with shared hyperparameters. We keep their module designs exactly and change only how the comparison is run. DANet, which uses parallel self-attention over positions and channels for segmentation, is excluded from classification because its quadratic position attention makes it a segmentation head, not a per-block gate. We treat it only as a reference point for the dense-prediction arm.
Three papers motivate the study by showing that training protocol can overturn architectural conclusions. ResNet strikes back raises vanilla ResNet-50 from 76.1% to 80.4% (A1, 600 epochs), 79.8% (A2, 300 epochs) and 78.1% (A3, 100 epochs), but never adds attention modules. Revisiting Point Cloud Shape Classification shows that modernizing PointNet++'s recipe erases its gap to newer networks. It also flags the practice of choosing checkpoints by test accuracy, which we avoid. Simple Local Attentions Remain Competitive shows that attention variants cluster on a proxy benchmark but separate on real tasks. That result motivates our dense-transfer arm: ImageNet may be the vision analogue of the Long Range Arena proxy, a benchmark on which attention variants look alike even though they differ on real tasks.
The under-tuned-baseline literature (Melis et al., Krichene, Hegde et al., Granero Moya et al., as summarized in the file's cluster on illusory progress) constrains the design. It rules out shared-hyperparameter comparisons and forces an equal tuning budget per arm.
Research Program
Backbones. We use ResNet-50, which lets us compare with CBAM, SCSA and ResNet strikes back, and MobileNetV2, which lets us compare with Coordinate Attention.
Modules. Each backbone is paired with seven attention settings: none, SE, CBAM channel-only, CBAM spatial-only, full CBAM, Coordinate Attention, and SCSA.
Recipes. Four recipe variants cross with the modules:
- the original 2015-style recipe;
- the A3 recipe;
- A3 with CutMix and Random Erasing removed;
- A3 with Mixup removed.
The third variant is the key manipulation, because it removes exactly the augmentations that delete image regions. The fourth separates region deletion from label mixing in general.
Seeds and schedule. Every cell gets three seeds. A3's 100-epoch schedule keeps the full grid affordable. The two most informative module settings plus the no-attention baseline are then re-run under A2 to confirm that conclusions hold at the longer schedule.
Tuning budget and model selection. Each arm receives the same tuning budget, one fixed-size random search over learning rate, weight decay and stochastic-depth rate. The search is scored on a held-out slice of the ImageNet training set, and the official validation set is touched once per final model.
Mechanistic probe. Alongside accuracy, we log for every spatial gate how far its mask departs from spatially uniform: the variance of the mask across positions, averaged over validation images and reported per network stage. The substitution hypothesis predicts that this variance shrinks when region-deleting augmentations are on. Channel gates get the analogous statistic across channels.
Baselines
Naive reference points. The "do nothing" reference is vanilla ResNet-50 and MobileNetV2 under each recipe with no attention. Under A3 this is also the strongest existing method for the backbone question. ResNet strikes back reports 78.1% top-1 on ImageNet-val for ResNet-50 at A3, 79.8% at A2 and 80.4% at A1. These are the numbers any attention module must beat on the same backbone. Our own re-run of the vanilla network gets the same tuning budget and seeds as every attention arm.
Strongest attention competitor. This is plain SE, the channel-only gate that CBAM and Coordinate Attention measured themselves against. It is the simplest module that could explain all of dual attention's benefit. We implement it in the same codebase, at the same insertion points, with the same tuning budget. The comparison with SE decides whether the spatial half earns its cost.
Mechanism ablation: static gates. For each spatial gate we train a version whose mask is a learned, input-independent parameter map of the same shape. For each channel gate we train a version with a learned per-channel constant. This separates "adding a few parameters and a multiplicative path" from "attending to the input." A module beats its static twin only if the gap exceeds the seed spread on paired seeds.
Mechanism ablation: parameter-matched backbone. A ResNet-50 widened slightly to match CBAM's roughly 0.4M extra parameters tests whether capacity alone explains the gains.
What counts as a win. A module "beats" a baseline if its mean over three paired seeds exceeds the baseline's by more than twice the pooled seed standard deviation. The substitution hypothesis is supported if the spatial-only gain under full A3 is less than half its gain under A3 without CutMix and Random Erasing, while the SE gain changes by less than that.
Why not just…?
Why not just use the modern recipe and skip attention entirely? This is the most serious alternative, and it may simply win: A3 alone reports 78.1%, above SCSA's 77.49% under its older recipe. We do not argue around it. The vanilla-under-A3 arm is the primary baseline, and "no module beats it beyond noise" is a publishable outcome. What this alternative cannot tell us is which component of the recipe replaces which kind of attention. Our augmentation ablations recover that from the same runs.
Why not just use SE? The CBAM–SE gap reported in the file is only 0.48 points of error, and the Coordinate Attention–SE gap is 0.9 points. Both are small enough to vanish under tuning. The SE arm, together with the CBAM channel-only and spatial-only splits, tests this directly. If SE matches every dual module under A3 and A2, we will report that dual attention is not justified for classification. The dense-prediction arm then asks whether spatial gating earns its keep only where the output is itself spatial.
Why not just compare published numbers, or add capacity instead? Published numbers mix recipes, which is exactly the confound this study removes. Extra capacity is tested by the width-matched backbone and the static-gate twins, so any claimed benefit of input-dependent attention must survive both.
Proposed Experiments
Decisive classification check. This is the recipe-by-module interaction on ImageNet-1K for ResNet-50, specifically the spatial-only CBAM gain with and without CutMix and Random Erasing, paired with the mask-variance probe. If the gain survives intact and the masks stay non-uniform, the substitution hypothesis is false, and spatial attention adds something that region deletion does not.
Dense-transfer check. Every A3-pretrained ResNet-50 checkpoint is fine-tuned with one fixed schedule:
- on ADE20K with UperNet, following SCSA's setup;
- on COCO with Faster R-CNN, following CBAM's setup.
We compute the rank correlation between each module's ImageNet gain and its dense gain. SCSA reports +0.94 mIoU on ADE20K and +1.7 AP on COCO under old recipes, and CBAM reports +1.7 mAP on COCO. We found no reported result on Lacuna for any of these modules on ADE20K or COCO under a modernized pretraining recipe, which is the gap this arm fills. A low or negative rank correlation would show that ImageNet is a misleading selection benchmark for spatial attention.
Cost measurement. Measured inference throughput on a single GPU is reported for every module. This lets any surviving accuracy gain be weighed against the memory-access overhead that SCSA acknowledges.
References
- CBAM: Convolutional Block Attention Module
- Coordinate Attention for Efficient Mobile Network Design
- SCSA: Exploring the synergistic effects between spatial and channel attention
- Dual Attention Network for Scene Segmentation.
- ResNet strikes back: An improved training procedure in timm
- Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline
- Simple Local Attentions Remain Competitive for Long-Context Tasks