Second-order pooling summarizes an image region by how its features vary together (a covariance matrix) instead of by their average strength. It has a reputation for robustness. The classical region covariance descriptor is unchanged under global brightness shifts, and in ultrasound a multi-scale second-order pooling layer pulled a gallbladder-cancer classifier away from shadows and speckle and toward tissue structure. That result, however, was measured only on a single hospital. The medical domain-shift literature, meanwhile, keeps finding that networks fail at new hospitals because they latch onto scanner-specific texture. These two observations conflict. In deep networks, the channel-by-channel covariance of feature maps is close to what neural style transfer calls "style," the texture statistics that scanners and stains change. Simply swapping average pooling for covariance pooling could therefore make a model more sensitive to scanner shift, not less.
This note proposes to resolve that conflict by splitting the covariance. Add pixel coordinates to the deep features before pooling, as classical region covariance descriptors do. The resulting matrix then has two informative blocks. One relates features to position: where in the region each feature fires, which reflects anatomy and layout. The other relates features to each other: which textures co-occur, which reflects style. Our claim is that hospital and scanner identity concentrate in the feature–feature block and in raw variances, while diagnostic information survives in the coordinate–feature block and in scale-free correlation coefficients. A pooling layer that keeps only the stable parts should generalize to unseen hospitals without hand-designed augmentation. The proposal measures this split directly before building on it.
Core Claim
Let $f_i \in \mathbb{R}^d$ be the deep feature at location $i$ of a region and $p_i = (x_i, y_i)$ its normalized coordinates (three coordinates for volumes). Write the covariance of the stacked vector $[p_i; f_i]$ in block form, and convert each block to correlations by dividing by the standard deviations:
$$ R = \begin{pmatrix} R_{pp} & R_{pf} \\ R_{fp} & R_{ff} \end{pmatrix}, \qquad R_{ab} = \mathrm{diag}(\Sigma_{aa})^{-1/2}\, \Sigma_{ab}\, \mathrm{diag}(\Sigma_{bb})^{-1/2}. $$
Here $\Sigma_{ab}$ is the sample covariance between coordinate set or feature set $a$ and set $b$. $R_{pp}$ is fixed by the grid and carries no information. $R_{pf}$ says how each feature's activation is arranged in space. $R_{ff}$ says which features co-occur. Dividing by the diagonal removes per-channel gain and contrast, the deep analogue of the brightness invariance of Tuzel et al..
The testable claim has two parts. First, a linear probe trained to predict the source hospital will find much more information in the variances and in $R_{ff}$ than in $R_{pf}$. A probe for the diagnostic label will show the opposite ordering, or at least a much smaller gap. Second, a pooling layer we call anatomy-anchored correlation pooling outperforms both average pooling and full covariance pooling on held-out hospitals and scanners. It outputs $R_{pf}$ plus off-diagonal $R_{ff}$, the latter passed through the smooth AsinhE power normalization of A Deeper Look at Power Normalizations and randomized during training.
Relation to Prior Work
Three papers motivate the gap. Surpassing the Human Accuracy: Detecting Gallbladder Cancer from USG Images with Curriculum Learning shows that multi-scale second-order pooling counters texture bias in ultrasound (91.0% versus 76.2% for ResNet50 on GBCU), but its evaluation is entirely in-distribution. Complex Style Image Transformations for Domain Generalization in Medical Images shows that prostate and cardiac MRI segmenters collapse on unseen scanners because they overfit to scanner texture. AMOS reports a drop of more than 20% mDice on MRI from unseen scanners. Together these say second-order statistics might help, and that texture is the enemy. They do not say which second-order statistics are texture.
Three papers supply the method. Global Second-Order Pooling Convolutional Networks gives the covariance-based recalibration block and its channel reduction. It pools the full feature covariance, with no coordinates and no domain analysis, and we test that design as a baseline. A Deeper Look at Power Normalizations gives coordinate augmentation and differentiable power normalization that avoids unstable eigendecompositions. The classical descriptors of Tuzel et al. and Tabia et al. give the coordinate-plus-feature construction and its scale independence.
Three papers use covariance for domain problems in ways that differ from ours. Domain Generalization with Correlated Style Uncertainty computes a covariance across the mini-batch of per-channel means and standard deviations and samples correlated style perturbations. Its classifier still pools first-order features. We work within each image, across space, and change the representation itself. Domain Generalization via Domain-based Covariance Minimization finds kernel projections that make domain labels unpredictable, on tabular data. We borrow its "measure domain predictability" logic as a diagnostic, but we do not learn a projection from the few available hospitals. Out-of-Distribution Robustness via Targeted Augmentations shows that with few training domains this kind of learned invariance is underspecified. Efficient Multi-Domain Learning by Covariance Normalization whitens and recolors activations per known domain, which presupposes access to the target domain.
Two papers constrain the design. WILDS finds that invariance objectives such as CORAL, which itself matches feature covariances across domains, rarely beat plain training on real shifts. The underspecification argument of Targeted Augmentations implies that removing only the handful of directions that separate training hospitals will not suffice. Our structural split is therefore fixed by construction, not learned from domain labels.
Research Program
The first stage is diagnostic and needs no new architecture. Take ERM-trained backbones on Camelyon17-WILDS, the single-source prostate MRI setting of CompStyle, and AMOS MRI grouped by scanner. Extract region features at several depths and compute five descriptors: average pooling, raw variances, $R_{ff}$, $R_{pf}$, and the full covariance. For each descriptor, train two linear probes on frozen features. One predicts the hospital or scanner; the other predicts the label, using tumor status for pathology and organ identity per region for segmentation. Because the descriptors differ greatly in size, report probe accuracy as a function of a fixed random-projection dimension. This stage either confirms the block-wise split or refutes it, and the method is only worth building if it is confirmed.
The second stage builds anatomy-anchored correlation pooling. For classification it replaces global average pooling. For segmentation it replaces the covariance in a GSoP-style recalibration block inside a 3D U-Net, computed on local windows so that coordinates are window-relative. A $1\times1$ convolution reduces channels to 64–128 before pooling, as in GSoP. During training, the $R_{ff}$ block is randomly attenuated or perturbed so the network cannot rely on it alone. The perturbation is generic and uses no modality-specific stain or artifact model.
We make one prediction up front. In 96-pixel pathology patches, spatial layout carries less diagnostic information than in abdominal or prostate volumes. We therefore expect $R_{pf}$ to matter most on MRI segmentation, and gains on Camelyon17 to come mainly from correlation normalization. If the reverse appears, our account of why the method works is wrong even if the numbers improve.
Baselines
The naive reference is ERM with global average pooling, the standard WILDS protocol. Model-Based Domain Generalization reports 73.3% out-of-distribution accuracy for ERM on Camelyon17-WILDS, and WILDS reports a fall from about 93% to about 70%. We rerun ERM with our seeds and backbone instead of quoting it.
The strongest existing competitor on Camelyon17 is CSU. Domain Generalization with Correlated Style Uncertainty reports 95.1% accuracy; Model-Based Domain Generalization reports 94.8% and Targeted Augmentations reports 90.5% with stain jitter. It is unclear from the file whether CSU used the official WILDS hospital split. We will therefore rerun CSU, stain jitter, and our method under the WILDS split, with the same backbone, epochs, and hyperparameter search budget, and with model selection on the WILDS out-of-distribution validation hospital. On prostate MRI the reference is CompStyle, which keeps Dice at roughly 64–77% on the hardest unseen sets where baselines fall to about 24–29%. On AMOS cross-scanner MRI, the file contains only the >20% drop; no reported result found on Lacuna for a domain-generalization method there.
Beating the field means the following. On Camelyon17, we must match the rerun CSU or targeted augmentation within seed variance without using stain-specific transforms, and exceed them when combined with them. On prostate MRI and AMOS MRI, we must improve worst-domain Dice over CompStyle at equal training cost.
The ablations isolating the key ingredient are full-covariance pooling as in GSoP, $R_{ff}$ only, $R_{pf}$ only, covariance versus correlation normalization, and the method with and without $R_{ff}$ perturbation.
Why not just…?
Why not just insert GSoP or multi-scale second-order pooling and evaluate out-of-distribution? This is the obvious experiment, and it is our central ablation. Our reason for doubting it is that full deep covariance includes exactly the texture co-occurrence that scanners alter. If full-covariance pooling matches our split version on held-out hospitals, the block decomposition adds nothing, and we will report that.
Why not just use instance normalization or per-channel standardization with average pooling? That removes per-channel scale, which is part of what correlation normalization does, and it is cheap. It does not remove co-occurrence statistics, and it discards spatial arrangement. Instance normalization plus average pooling is included at matched compute. If it equals our method, the gain came from scale removal alone and the covariance machinery is unnecessary.
Why not just pool spatial moments of each feature map, since $R_{pf}$ is essentially per-channel centroid information? This is a real threat. $R_{pf}$ is close to normalized first spatial moments, and a "where does each channel fire" pooling might capture the whole benefit. We include spatial-moment pooling as an explicit baseline. If it matches our method, the honest conclusion is that robustness comes from layout pooling rather than second-order statistics, and that is still a useful finding.
Why not just use targeted augmentation or CSU? On pathology they are strong: 90.5% and 95.1% as reported. Their limits are elsewhere. Targeted augmentation needs a physical model of the artifact, which is unavailable for MRI scanner differences. CSU perturbs first-order statistics and leaves the representation unchanged. The prostate and AMOS experiments test whether a structural pooling choice transfers where hand-built augmentation cannot. The combination experiments test whether the gains add up.
Why not just use an invariance objective such as CORAL or Group DRO? WILDS found such objectives failed to beat ERM on 8 of 10 real shifts, and with few hospitals the learned invariance is underspecified. We include CORAL for completeness but do not expect it to be competitive.
Proposed Experiments
The decisive first check is the probe study on Camelyon17-WILDS, prostate MRI, and AMOS MRI. It passes if, at matched dimension, hospital or scanner probe accuracy on $R_{pf}$ is clearly closer to chance than on $R_{ff}$ and the variances, while label probe accuracy on $R_{pf}$ plus off-diagonal $R_{ff}$ stays within a few points of full covariance. If domain identity is equally decodable from every block, the premise fails and we stop.
The main evaluation trains on the source domains with five seeds. On Camelyon17-WILDS we report out-of-distribution accuracy on the test hospital and the gap to in-distribution accuracy. On prostate MRI we report mean and worst Dice across CompStyle's seven external datasets. On AMOS we train on the seen MRI scanners and report mDice on the unseen ones. On ACDC with synthetic spike and ghosting corruptions, we test whether correlation normalization helps against artifacts that change variances. We also report GFLOPs, since GSoP adds 1.6–1.7×, and we expect our reduced-channel version to cost less.
The final check is a falsification test of the mechanism. We re-measure hospital-probe accuracy on the trained model's pooled descriptor and correlate it with out-of-distribution error across all variants. If our account is right, lower domain decodability should predict better transfer, and the $R_{pf}$ contribution should be larger on MRI than on pathology.
References
- Pedestrian Detection via Classification on Riemannian Manifolds
- A Deeper Look at Power Normalizations
- Surpassing the Human Accuracy: Detecting Gallbladder Cancer from USG Images with Curriculum Learning
- Complex Style Image Transformations for Domain Generalization in Medical Images
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Global Second-Order Pooling Convolutional Networks
- Human Detection via Classification on Riemannian Manifolds
- Covariance Descriptors for 3D Shape Matching and Retrieval
- Domain Generalization with Correlated Style Uncertainty
- Domain Generalization via Domain-based Covariance Minimization
- Out-of-Distribution Robustness via Targeted Augmentations
- Efficient Multi-Domain Learning by Covariance Normalization
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Model-Based Domain Generalization