Most image classifiers summarize a grid of feature vectors by averaging them. The average throws away which features occur together, and the fine-grained recognition literature has shown that keeping the covariance (the pairwise co-occurrence of feature channels) recovers much of what is lost. Separately, models that read images at their native shape and resolution, rather than squashing them into squares, avoid the distortion and cropping losses that hurt elongated or off-center content. These two improvements look compatible, but they are not obviously so. A native-shape model produces a different number of tokens for every image and every compute budget. An average is a well-behaved statistic at any token count. The eigenvalue spectrum of a sample covariance is not: its spread depends on the ratio of channels to samples. The very small eigenvalues that carry fine-grained cues are the ones most contaminated by this finite-sample noise.
We propose to measure this failure and then fix it with a pooling head built for native-shape inputs. The head splits the covariance into a within-neighborhood part, computed inside fixed-size square windows laid on the native 2D token grid, and a between-neighborhood part, computed over window summaries. Each part is denoised using its own effective sample size before matrix square-root normalization. The split also addresses the local-versus-global tension. The within term records whether features co-occur in the same small region, such as blue and feathered on the same patch. The between term records how regions relate across the whole image. Because the windows are defined in token units, neither term depends on whether the image is square, panoramic, or tall.
Core Claim
Our claim is that second-order pooling, moved naively onto variable-token-count, native-aspect-ratio backbones, loses robustness to the token budget. A head trained at one budget will degrade when evaluated at another, and the degradation will concentrate in the small-eigenvalue directions. Averaging does not suffer this, because the mean is unbiased at any sample size.
The reason comes from random matrix theory. For $N$ tokens of dimension $d$ with isotropic noise of variance $\sigma^2$, the sample covariance's eigenvalues spread over roughly
$$[\sigma^2(1-\sqrt{\gamma})^2,\ \sigma^2(1+\sqrt{\gamma})^2],\qquad \gamma = d/N,$$
the Marchenko-Pastur interval. The noise floor therefore moves whenever $N$ changes, and so does how much a fixed square-root normalization amplifies or flattens. Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias supplies this principle. It shows that a weight matrix's shape skews its spectrum and that comparing spectra requires fixing the shape, which their method does by subsampling square submatrices. That work is about weight matrices in language and CIFAR models, not visual feature covariance. Our contribution is to show the same effect in pooling and to design around it.
The fix uses the law of total covariance over $K$ windows $w$ with token sets $\mathcal{W}_w$, window means $m_w$, and global mean $\mu$:
$$\Sigma = \underbrace{\tfrac{1}{N}\textstyle\sum_w \sum_{i\in\mathcal{W}_w}(x_i-m_w)(x_i-m_w)^\top}_{\Sigma_{\text{within}}} + \underbrace{\tfrac{1}{N}\textstyle\sum_w |\mathcal{W}_w|\,(m_w-\mu)(m_w-\mu)^\top}_{\Sigma_{\text{between}}}.$$
Standard global covariance pooling uses only the sum $\Sigma$, so it mixes local co-occurrence with layout-scale variation and treats both as having $N$ samples. In fact the between term has only $K-1$ degrees of freedom. On a small budget or an extreme aspect ratio, $K$ is small and that term is mostly noise. Our head proceeds in four steps. It shrinks each term's eigenvalues below its own noise edge, using $\gamma_{\text{within}} = d/(N-K)$ and $\gamma_{\text{between}} = d/(K-1)$. It applies matrix square-root normalization to each term separately. It concatenates the two upper triangles. It does all of this with masks, so that images packed into one sequence never contribute to each other's statistics.
Relation to Prior Work
Three papers motivate the gap by showing that the spectrum is the crux of second-order pooling, all under fixed square inputs:
- Deep CNNs Meet Global Covariance Pooling introduces matrix-power-normalized covariance pooling and its fast Newton-Schulz variant. It replaces averaging and reports a ResNet-50 ImageNet gain of over 2.5%.
- Revitalizing SVD for Global Covariance Pooling shows that Newton-Schulz over-flattens the spectrum and replaces it with a third-order Halley iteration.
- On the Eigenvalues of Global Covariance Pooling for Fine-Grained Visual Recognition shows that small eigenvalues carry the fine-grained cues and amplifies them with a parameter-free branch.
None of the three varies the token count, so none can see that the noise floor they normalize against is a function of $N$. The third paper also constrains our design. Its finding that truncating small eigenvalues collapses fine-grained training means our shrinkage must be calibrated to the noise edge rather than act as a hard cutoff. We test that directly.
Tuning Pre-trained Model via Moment Probing is the closest local-plus-second-order method. It computes cross-covariance between adjacent channel groups and then mixes the result with a small 2D convolution. Its covariance is still computed over all spatial positions, so locality enters only after spatial information has been discarded. We put locality inside the statistic. Patch n' Pack: NaViT supplies the input side: packed variable-length sequences, factorized $x/y$ position embeddings, and a single checkpoint evaluated across token budgets. It ends in masked average pooling, which is the first-order bottleneck we replace.
Aspect-Ratio-Preserving Multi-Patch Image Aesthetics Score Prediction motivates the aspect-ratio regime. It reports that its error reduction on extreme-ratio images was up to 7.9 times larger than on ordinary 3:2 or 4:3 photos. Its patch fusion, however, is first-order. Invariant Deep Compressible Covariance Pooling handles discrete rotations and reflections of square inputs with covariance pooling. Continuous aspect-ratio change is outside its scope, which is our setting.
Research Program
Diagnosis. The first part is a controlled measurement. We take a native-resolution ViT-B/16 with NaViT-style factorized position embeddings, fine-tuned on ImageNet-1K with packing. We feed it the same image at several token budgets and at native versus square-resized geometry. For each setting we record three things: the covariance spectrum, the spectral flatness index used by the Halley-SVD paper, and the fraction of eigenvalues inside the predicted noise interval. The prediction is specific. As $N$ falls, the lower eigenvalues should crowd toward the noise edge. The flatness of the square-root-normalized matrix should then drift with $N$ even when the image content is held fixed.
Construction. The second part builds the masked within/between head. The windows are 4×4 tokens on the native grid; partial windows at borders are weighted by their size. A 1×1 projection reduces features to $d=256$, following the covariance pooling literature. The masked statistics are computed with segment-wise matrix products over packed sequences. Noise-edge shrinkage is followed by Newton-Schulz or Halley square-root normalization, so the head reuses existing normalizers rather than competing with them.
Robustness to budget. The third part tests budget and aspect-ratio robustness against the existing second-order heads, in two settings. In the frozen-backbone probing setting, which is cheap and comparable to Moment Probing, heads are trained at one budget and evaluated across the whole budget range. In the full fine-tuning setting, the backbone is trained with randomly sampled budgets, as NaViT does.
Baselines
Naive reference. NaViT's own masked average pooling with a linear head, and the same with a two-layer MLP head of matched parameter count. This is the first-order reference that every second-order head must beat. Because the mean does not depend on sample size, we expect it to have the flattest accuracy-versus-budget curve. The MLP variant separates "more head capacity" from "second-order statistics."
Strongest existing second-order heads. Global covariance pooling with Newton-Schulz square-root normalization (Deep CNNs Meet Global Covariance Pooling), with Halley iterations (Revitalizing SVD), and with the Scaling Eigen Branch added (On the Eigenvalues of GCP). Each is ported to packed sequences with exact masking, so that the only difference from our head is the estimator. Moment Probing's adjacent-group cross-covariance head (Tuning Pre-trained Model via Moment Probing) is the strongest local-plus-second-order competitor in the probing setting.
Reported numbers. For reference on square inputs, the file reports:
- Moment Probing beats linear probing by an average of 9.24% top-1 across CUB-200-2011, Oxford Flowers, and Stanford Cars.
- Halley-SVD beats Newton-Schulz by an average of 1.2% across CUB-200, Stanford Dogs, and Stanford Cars.
- The Scaling Eigen Branch adds an average of 1.6% to covariance pooling on fine-grained benchmarks.
- MPN-COV reaches 88.7% on CUB-200-2011.
For native-aspect-ratio fine-grained classification with any second-order head, no reported result was found on Lacuna, so we will establish the baseline numbers ourselves.
Fair comparison. Every head gets the same backbone checkpoint, the same packed data, the same reduced dimension $d$, the same number of epochs, and the same hyperparameter search of twenty trials over learning rate and weight decay. Wall-clock time is reported alongside accuracy.
Ablations. These isolate the key ingredients. The within-only and between-only variants test whether the split matters. Our split without shrinkage, and global covariance with noise-edge shrinkage but no split, separate the two ideas. Shrinkage with a fixed $\gamma$ taken from the training budget tests whether per-image calibration matters.
What counts as a win. Beating the strongest baseline means two things together. First, matching or exceeding Halley-SVD with the Scaling Eigen Branch at the training budget. Second, losing clearly less accuracy than it when evaluated at budgets half and double the training budget, with the gap concentrated on the extreme-aspect-ratio test slices.
Why not just…?
Why not just use a standard shrinkage estimator, such as adding a scaled identity or Ledoit-Wolf shrinkage, to existing covariance pooling? This is the most serious threat. Noise-edge shrinkage is a form of shrinkage, and if a per-image Ledoit-Wolf global covariance matches our head, then the within/between split adds nothing. We include exactly this as an ablation, using global covariance with sample-size-calibrated shrinkage. There are two reasons we expect the split to matter. Shrinkage on the mixed $\Sigma$ applies one $\gamma = d/N$ to a matrix whose between-window component effectively has only $K-1$ samples, so it under-shrinks that component. And shrinkage cannot restore the spatial locality that the blue-bird example requires. If the ablation ties our head, we will report the contribution as the diagnosis plus calibrated shrinkage and drop the split.
Why not just keep a fixed token budget while preserving aspect ratio? NaViT can resize each image to a constant number of tokens without distorting its shape, which removes the variation in $N$ for any single deployment. That works if one budget is all you ever need. But it gives up the main practical property of native-resolution models, one checkpoint evaluated anywhere on a speed-accuracy curve. It also does not fix the between-window term, because $K$ along each axis still changes with aspect ratio at a fixed total budget. Our aspect-ratio-sliced evaluation at a fixed budget tests that second point directly.
Why not just train the existing covariance head with random token budgets so it learns invariance? This could work, and we treat it as a decisive comparison. Newton-Schulz and Halley heads will be trained with NaViT's random budget sampling under the same compute as our head. If they become as flat across budgets as ours, the estimator is unnecessary for fine-tuned models. Its value would then be limited to the frozen-probing regime, where the head cannot reshape the features, and we would say so.
Why not just keep averaging, since native-shape attention already mixes local and global context? The file's evidence argues against this for fine-grained tasks. Moment Probing's 9.24% average margin over linear probing, and MPN-COV's gain of over 2.5% over averaging on ImageNet, both show that the mean discards usable signal. Whether the same holds after native-shape attention is exactly what our averaging-plus-MLP baseline measures.
Proposed Experiments
Datasets. Fine-grained evaluation uses CUB-200-2011, Stanford Cars, Stanford Dogs, and FGVC Aircraft, read at native aspect ratio. These datasets contain many elongated subjects: limousines, aircraft fuselages, long-tailed birds. Each one is also run under the conventional square resize, so the effect of geometry and the effect of the head can be separated. General-purpose evaluation uses ImageNet-1K. Robustness uses ImageNet-A, where NaViT's avoidance of center cropping is reported to help, together with ImageNet-R and ImageNet-C. Remote-sensing transfer uses NWPU-RESISC45, whose wide scene layouts stress the between-window term.
Metrics. We report top-1 accuracy at the training budget. We report the area under the accuracy-versus-budget curve across a fourfold budget range, which is the headline robustness metric. We report accuracy on test images binned by aspect ratio. And we report the spectral flatness index of the normalized pooled matrix as a function of $N$.
First decisive check: diagnosis. If the flatness index of a Newton-Schulz head is insensitive to $N$ at fixed content, the Marchenko-Pastur argument does not apply here. We would then abandon the estimator and report the native-ratio fine-grained benchmark alone.
Second decisive check: robustness. A probing head trained at one budget must keep its fine-grained margin over masked averaging when evaluated at half that budget. The prediction is that the existing second-order heads lose most of their margin there and ours retains it.
Third decisive check: the answer to "why not just." Random-budget training of baseline heads, and calibrated shrinkage of global covariance, are each run under identical compute. Either one tying our head redefines the contribution in the way stated above.
Secondary question. We also ask whether the published fine-grained gains of covariance pooling grow or shrink once images stop being squashed. This tells the community whether part of those gains were compensating for resize artifacts.
References
- Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
- Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization
- Revitalizing SVD for Global Covariance Pooling: Halley’s Method to Overcome Over-Flattening
- On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognition
- Tuning Pre-trained Model via Moment Probing
- Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
- Aspect-Ratio-Preserving Multi-Patch Image Aesthetics Score Prediction
- Invariant Deep Compressible Covariance Pooling for Aerial Scene Categorization