When a convolutional network ends in global average pooling, which averages each feature map over all spatial positions into one number, it can accept images of any size. That flexibility has a cost when the evidence for the label is small, such as a lesion in a whole breast scan or a few brushstrokes on a large print. Each location's share of the gradient is one over the number of positions, so as the image grows, the evidence that decides the label contributes a vanishing fraction of the learning signal. One paper in the file shows the extreme case. A 3D ResNet trained directly on whole breast MRI volumes stayed at chance (AUROC 0.50). The same network reached 0.89 when it was first trained on small patches centred on annotated lesions and then moved to whole volumes Multi Scale Curriculum CNN for Context-Aware Breast MRI Malignancy Classification.
We argue that the curriculum in that result works for a specific reason: it keeps the evidence a large fraction of what gets pooled while the network is still untrained. If that is right, the curriculum does not need to change the data at all. It can instead change the pooling operator. Training starts with pooling that behaves like a max and so sends the full gradient to the strongest-responding location, then gradually relaxes to a plain average over the whole, unaltered image. This schedule needs no lesion annotations or crops. Because the final stage of training uses exactly the pooling and inputs seen at test time, it also avoids the train-test mismatch that data curricula risk Spatial Transformer Networks for Curriculum Learning.
Core Claim
We test two linked claims. The first is a diagnostic claim. The benefit of any context curriculum on fully convolutional networks should grow with the ratio of image area to evidence area. A curriculum should help little on tasks where the evidence is spread across the image, such as photo aesthetics, and a great deal where the evidence is a needle in a haystack. The second claim is the method: a curriculum over the pooling exponent recovers most of the patch-to-whole benefit without the annotations that the MRI work needs to build its easy stage.
We use generalized-mean pooling as the knob. For channel $c$ with activations $x_{c,u} \ge 0$ at positions $u$ in the spatial set $\Omega$, it is defined as
$$ y_c(p) = \Big(\tfrac{1}{|\Omega|}\textstyle\sum_{u \in \Omega} x_{c,u}^{\,p}\Big)^{1/p}. $$
Here $p=1$ gives average pooling and $p \to \infty$ approaches max pooling. The schedule anneals $p$ from a large value $p_0$ down to 1 over a fraction of training. Under average pooling, a region covering fraction $\rho$ of the feature map receives fraction $\rho$ of the gradient. Under large $p$, the gradient concentrates on the highest-responding positions. This concentration is what we expect to rescue optimization when $\rho$ is small.
We end at average pooling rather than staying at max for a stated reason. Max pooling sends gradient to one location per channel and discards diffuse context. In breast MRI, that context includes background tissue enhancement and the lesion's position relative to the whole breast, which is the very thing the MRI paper moved to whole volumes to capture.
Relation to Prior Work
Three papers supply the motivation. The MRI paper is the closest work. It uses a two-step data curriculum that moves from annotated patches to whole volumes, with adaptive average pooling so the classifier head is unchanged between stages. We keep its goal and backbone but replace the data staging with operator staging, and we drop the need for lesion centres.
SPP-net introduced pooling into fixed spatial bins so that inputs of any size produce a fixed-length vector Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. Its multi-size training alternates between two square resolutions with no ordering. It therefore supplies the unordered scale-mixing baseline we must beat.
van Noord and Postma showed that artist attribution on Rijksmuseum prints depends on fine, scale-specific detail Learning scale-variant and scale-invariant features for deep image classification. Their network with global average pooling over a Gaussian pyramid reached 82.11% mean class accuracy as an ensemble, against 79.98% for standard multi-scale training. They did not try ordered schedules, and their dataset gives us a second small-evidence domain.
The curriculum literature supplies both the method and its constraints. Self-paced learning lets the model admit harder samples as a loss threshold rises Self-Paced Learning for Latent Variable Models. We borrow its idea of a schedule that loosens over training, but we apply it to an operator rather than to samples. The spatial-transformer curriculum builds an easy-to-hard spectrum by progressively uncluttering digits. It reports 95.0% on Cluttered MNIST against 91.2% for data augmentation. It also constrains our design, because it stresses that the last training stage must match the raw test distribution, which our schedule satisfies by construction.
The aesthetics work supplies a negative control. MNA-CNN Composition-Preserving Deep Photo Aesthetics Assessment and MLSP Effective Aesthetics Prediction With Multi-Level Spatially Pooled Features both show that aesthetics depends on whole-image composition. On such data our diagnostic claim predicts that an annealed schedule gives little gain, and might even hurt.
Research Program
We first build a controlled testbed in which the evidence fraction $\rho$ is set directly. Cluttered MNIST digits are placed on canvases whose side grows from the standard size up to roughly eight times larger, with clutter density held fixed. A fully convolutional network with generalized-mean pooling is trained from scratch under the following pooling regimes:
- fixed average pooling;
- fixed max pooling;
- fixed $p$ learned jointly with the weights;
- the annealed schedule;
- the reverse schedule, which runs from average to max.
The same network is also trained under a data curriculum that grows random crops from small to full canvas. For each canvas size we record final accuracy and the training step at which loss first drops measurably below chance. If the dilution account is correct, the canvas size at which fixed average pooling collapses should follow the evidence fraction $\rho$, and the annealed schedule should push that collapse point outward.
We then move to the two real small-evidence domains: the RWTH breast MRI cohort used by the MRI paper (408 patients), and Rijksmuseum artist attribution at native high resolution. On both we train end-to-end on whole inputs. The schedule has two hyperparameters, the starting exponent $p_0$ and the annealing length. We tune them on a validation split and report sensitivity curves so the schedule's robustness is visible rather than hidden. Finally, we run AVA as the diffuse-evidence control, using binary accuracy and Spearman rank correlation (SRCC).
Baselines
The do-nothing reference is the same fully convolutional backbone trained on whole inputs with fixed average pooling and no curriculum. This corresponds to the Naive ResNet of the MRI paper, which reported AUROC 0.50 and accuracy 0.45. We rerun it under our training budget rather than relying on that figure, because its collapse may partly reflect training from scratch on a small cohort.
The strongest existing method on this question is the MRI paper's patch-to-whole curriculum, which reported AUROC 0.89 and accuracy 0.81 on the RWTH cohort, matching Retina U-Net at 0.89. We reimplement it with the same backbone, epochs and hyperparameter trial budget as our method. It uses lesion-centre annotations that our method does not. Our result counts as beating it if we match its AUROC within the cohort's bootstrap confidence interval while using no spatial annotations, and as a clear win only if we exceed that interval. On Rijksmuseum the reference points are the 82.11% ensemble and the 79.98% standard multi-scale training figure. The ensemble runs four networks at inference, so we compare against it at matched and at single-network inference cost.
The key ablation is fixed max pooling, and fixed learned-$p$ pooling, with no schedule at all. These isolate whether the ordering matters or only the pooling choice does. The reverse schedule tests whether moving from concentrated to diffuse pooling is specifically what helps.
Uniform multi-size jitter in the style of SPP-net, and an unannotated random-crop curriculum, represent the data-side alternatives. On Cluttered MNIST at its standard canvas we also compare against the spatial-transformer Incremental Difficulty result of 95.0%. Every method receives the same number of gradient steps and the same number of tuning trials.
Why not just…?
Why not just use max pooling, or a learned pooling exponent, from the start? This is the real threat. If fixed large $p$ matches the annealed schedule on MRI and Rijksmuseum, then the curriculum is unnecessary and the finding reduces to "use the right pooling operator for small evidence." We would report it that way. Our reason to expect a gap is that max pooling ignores diffuse context. We test that prediction directly by measuring AUROC on MRI cases where the radiological signal is diffuse, and by measuring whether the learned $p$ itself drifts toward 1 late in training.
Why not just crop around the evidence, as the MRI paper does? That approach requires knowing where the evidence is, and the MRI paper used annotated lesion centres to get it. Without annotations, random crops of a positive scan often miss the lesion, which gives the network wrongly labelled training examples. Our unannotated growing-crop baseline measures how much this costs.
Why not just start from an ImageNet-pretrained backbone? Pretrained features may already respond to small objects, which would weaken the dilution problem. We therefore run every regime from both random and pretrained initialization. We expect the curriculum gap to shrink with pretraining but not to disappear at large area ratios. If it does disappear, the method's scope is limited to training from scratch, and we would state that.
Why not just jitter scales randomly? In the Rijksmuseum study, standard multi-scale training trailed the ensemble (79.98% versus 82.11%), which suggests that mixing scales without order is not sufficient. Our matched-budget jitter baseline tests this on our own tasks.
Decisive Checks
Three outcomes would decide the proposal. The first is whether, on the Cluttered MNIST canvas sweep, the collapse point of fixed average pooling follows the evidence fraction $\rho$ and the annealed schedule moves it outward, while the reverse schedule does not. The second is whether, on breast MRI, annotation-free annealing comes within the confidence interval of the annotated patch-to-whole curriculum's 0.89 AUROC. The third is whether, on AVA, the gain over fixed average pooling is near zero. A near-zero AVA gain would confirm that the effect is about small evidence and not a general training improvement.
References
- Multi Scale Curriculum CNN for Context-Aware Breast MRI Malignancy Classification
- Spatial Transformer Networks for Curriculum Learning
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
- Learning scale-variant and scale-invariant features for deep image classification
- Self-Paced Learning for Latent Variable Models
- Composition-Preserving Deep Photo Aesthetics Assessment
- Effective Aesthetics Prediction With Multi-Level Spatially Pooled Features