A convolutional network sees the world through small windows. Each unit looks at a few neighboring pixels, so a feature that depends on something far away, like "is this red sphere left of the cube?" or "is this whole image uniformly blurred?", has to be pieced together slowly over many layers. The usual fixes are attention, large kernels, or recurrent sweeps. Attention compares every location with every other one, which is expensive. Large kernels and sweeps are poorly supported on ordinary hardware. Speech researchers solved a related problem long ago with a much cheaper tool. They summarize a variable-length signal with a few statistics, such as its mean, spread, asymmetry, tails, and extremes, and hand that summary to a classifier. We propose to bring this tool inside the network. At intermediate layers, each channel's activation map is summarized by a handful of statistics computed in a single pass, and that summary is copied back to every pixel, so each location sees a compact description of the whole image.
What makes this proposal more than "add skewness to Squeeze-and-Excitation" is the second family of statistics. Besides the moments of activation values ("how much, and how unevenly, is this feature present"), we compute moments of activation-weighted pixel coordinates ("where is this feature's mass centered, and how spread out is it"). Each pixel then receives its own offset from every channel's center of mass. This gives every location a linear-cost, matrix-multiply-free answer to "where is the thing that channel $c$ detects, relative to me." We think this answer is a large part of what attention buys on relational and dense-prediction tasks. Value moments and location moments should help on different tasks, and the core experiment is built to test exactly that split.
Core Claim
For each channel $c$ of a feature map with nonnegative activations $a_c(u)$ at pixel coordinates $u \in [-1,1]^2$, define the normalized spatial mass $p_c(u) = a_c(u)/\sum_{u'} a_c(u')$. We compute two kinds of statistics. The first is the value moments of $a_c$ over all pixels: mean, standard deviation, standardized skewness, standardized kurtosis, and maximum. The second is the location moments of $p_c$: the centroid and the spatial covariance,
$$\mu_c = \sum_u p_c(u)\,u, \qquad \Sigma_c = \sum_u p_c(u)\,(u-\mu_c)(u-\mu_c)^\top .$$
The broadcast step gives every pixel $u$ the value moments, which are the same everywhere, together with a position-dependent term: its offset $u-\mu_c$ and its normalized distance $(u-\mu_c)^\top\Sigma_c^{-1}(u-\mu_c)$ for each channel. These are compressed by a $1\times1$ convolution and added back to the feature map through a learned gate. The module costs $O(NC)$ for $N$ pixels and $C$ channels. It uses only sums of powers and coordinate-weighted sums, which is the reduction profile that ParCNetV2 notes edge accelerators handle well, in contrast to oversized kernels and affinity matrices.
We make three falsifiable predictions. Location moments will carry most of the gain on relational reasoning and localization. Higher-order value moments, meaning skewness and kurtosis rather than only mean and standard deviation, will carry the gain on image quality assessment, where distortions show up as heavy tails. On generic classification, the module will match second-order pooling at a fraction of its cost.
Relation to Prior Work
Four papers sit directly at this intersection, and each supplies one piece while leaving out another. Spatial Moment Pooling replaces final average pooling with the first four moments. It stabilizes skewness and kurtosis with layer normalization, and it raises NIMA's LIVE SRCC from 0.864 to 0.890. It supplies our stabilization method, but it operates only at the network output, never feeds the moments back into intermediate layers, and ignores location. Global Second-Order Pooling shows that intermediate statistical recalibration beats terminal-only pooling, with 21.19% ImageNet top-1 error versus 23.85% for ResNet-50. It motivates the intermediate placement, but it is limited to covariance and costs 1.6 to 1.7 times the GFLOPs. Its position-wise "where" branch also needs an $N\times N$ covariance, whereas we get "where" from $O(C)$ centroids. Broadcasting Convolutional Network broadcasts a global vector to every pixel after appending coordinate channels, reaching 97.7% on CLEVR. It supplies the broadcast design, but its only summary is max-pooling, a single order statistic. That gives one location per channel and no spread, and nothing about the distribution of values. Statistically-Motivated Second-Order Pooling constrains our design from the other side. It argues that classifier heads want Gaussianized inputs and applies a Wilson-Hilferty transform. Kurtosis features are deliberately non-Gaussian, so we test both raw standardized moments and a transformed version rather than assuming one answer.
Two further works frame the cost argument. Compact Generalized Non-local Network proves that bilinear pooling is a special case of non-local attention. A broadcast moment module can therefore be read as a deliberately restricted, cheap point inside the attention family, and the question becomes how much of attention's benefit survives the restriction. CCNet reduces attention to $O(N\sqrt{N})$ through criss-cross paths, but it still needs two passes and gather operations.
The speech side supplies the vocabulary and a known weakness. Functionals over frame-level descriptors are robust to local noise (Meghjani et al.). Their discriminative content often lives in skewness, kurtosis, and extremes (Orozco-Arroyave et al., Black et al.). But they discard order, so a rising and a falling contour get identical statistics (Wöllmer et al. responded with recurrence). Location moments are our answer to that weakness in two dimensions. They are the image analogue of the regression-slope functionals in speech, because they keep coarse geometry without keeping the full sequence.
Research Program
We will build the module in three variants that share all code. The first uses value moments only, in two sub-variants: mean and standard deviation alone, or the full set with skewness, kurtosis, and max. The second uses location moments only. The third combines both. Each variant is placed after every stage of a ResNet backbone, following GSoP's intermediate placement. Skewness and kurtosis are computed on channel-standardized activations with an epsilon floor and then layer-normalized, as in Spatial Moment Pooling. Activations are passed through a softplus before forming $p_c$, so the spatial mass is well defined.
Alongside accuracy, we will measure effective receptive field size by gradient mapping at stage 3, to check whether the module actually widens it. We will also measure wall-clock latency on a GPU and on a mobile CPU through a standard runtime. A diagnostic probe tests whether stage-3 features can linearly decode the position of a distant object, comparing the backbone with and without the location branch. This probe is the most direct test of the "where" claim.
Baselines
The naive reference is the unmodified ResNet backbone trained with an identical recipe. For calibration, Global Second-Order Pooling reports ResNet-50 at 23.85% ImageNet top-1 error and a CIFAR-100 baseline at 24.33% error.
The strongest statistical competitor is GSoP itself, which reports 21.19% ImageNet top-1 error and 18.58% on CIFAR-100 for GSoP-Net2. We will rerun GSoP-Net1, which uses intermediate blocks and an ordinary final pooling layer, in our own pipeline. Comparing it with our modules under the same placement isolates the intermediate operator. We count it as beaten if we match its error within 0.3 points at no more than 1.1 times the backbone's GFLOPs, or exceed it outright.
The channel-attention baseline is Squeeze-and-Excitation, which uses the mean only. GSoP reports SE at 23.29% and CBAM at 22.66% ImageNet top-1 error. Both will be rerun at equal placement and equal reduction ratio.
The key ablation is a mean-plus-standard-deviation recalibration module with no location branch and no higher moments. It isolates whether anything beyond second-order channel statistics matters. A second ablation replaces location moments with BCN-style coordinate channels plus global max-pooling.
For relational reasoning, the strongest existing method is BCN with multiRN, which reports 97.7% CLEVR accuracy from raw pixels and a sixfold reduction in Scaled-MNIST localization error. We will swap only the global descriptor inside multiRN. Beating it means higher CLEVR accuracy, or equal accuracy with fewer parameters.
For dense prediction, the strongest baseline is CCNet, which reports 81.9% mIoU on the Cityscapes test set and 45.76% on the ADE20K validation set. We will rerun it on the same backbone, crop size, and schedule. Our claim here is a better accuracy-memory trade-off, not higher mIoU.
For quality assessment, the strongest baseline is HyperIQA-SMP(2) at 0.909 SRCC on Koniq-10k, together with the NIMA ablation on LIVE (SAP 0.864, SMP(4) 0.890 SRCC). CSIQ will serve for cross-dataset tests; no reported result found on Lacuna for SMP on CSIQ.
All methods get the same data, epochs, augmentation, and a hyperparameter sweep of the same size (learning rate and weight decay, three seeds each). Reported numbers are used only to check that our reruns reproduce the published baselines.
Why not just…?
Why not just add mean-and-standard-deviation recalibration to Squeeze-and-Excitation? This is the most serious threat. On ImageNet it may capture nearly all the gain that value moments provide, and the evidence that higher moments help is thin: SMP's LIVE gain from second to fourth order is only 0.886 to 0.890 SRCC. We treat this baseline as the default hypothesis. If it ties our value-moment variant everywhere except IQA, we will report that, and the contribution narrows to the location branch, which mean-and-standard-deviation pooling cannot supply because it is permutation-invariant over pixels.
Why not just use BCN's coordinate channels with max-pooling? Max-pooling coordinate-augmented features gives one location per channel. That location is the most active pixel, which is brittle when a channel fires on several objects or on texture. Centroid and spread degrade gracefully and are differentiable everywhere. The CLEVR and Scaled-MNIST swap inside multiRN settles this directly. If max-pooling ties, location moments add nothing over BCN.
Why not just use soft-argmax or spatial softmax, which already compute activation centroids? These compute the same first location moment and are a fair alternative; we found no paper on Lacuna evaluating them as intermediate broadcast modules. Our location-only variant without $\Sigma_c$ is exactly this baseline. The comparison tells us whether the spread term and the per-pixel offset broadcast matter, or whether the centroid alone does the work.
Why not just use non-local or criss-cross attention? Attention is more expressive, and by the CGNL result it subsumes second-order pooling. So we do not expect to beat CCNet on mIoU. The question is cost. Dense non-local blocks are quadratic, and CCNet reports needing 11 times less memory than them while still requiring two passes. If our module recovers most of CCNet's gain over the backbone at a fraction of its latency on a mobile CPU, the trade is worth having. If it recovers little, that is a clean negative result about how much of attention is "where" rather than content-dependent routing.
Why not just go deeper? The effective receptive field grows much more slowly than depth suggests (Li et al.). We will nonetheless include a deeper backbone with matched FLOPs as a control, for example ResNet-101 against ResNet-50 with our module.
Decisive Checks
The proposal stands or falls on a crossover pattern. On CLEVR, Sort-of-CLEVR, and Scaled-MNIST, the location branch should account for most of the improvement over the backbone, and the value-only variants should do little. On Koniq-10k and LIVE, the opposite should hold, with skewness and kurtosis beating the mean-and-standard-deviation ablation beyond seed variance. On ImageNet and CIFAR-100, the combined module should approach GSoP-Net1 at near-backbone cost, and on Cityscapes it should give most of CCNet's gain at lower memory. The position probe should show distant-object position becoming linearly decodable at stage 3 only when the location branch is present. If the location branch fails on relational tasks, or if the mean-and-standard-deviation ablation ties everything everywhere, the central claim is wrong. We will report it that way.
References
- ParCNetV2: Oversized Kernel with Enhanced Attention
- Spatial Moment Pooling Improves Neural Image Assessment
- Global Second-Order Pooling Convolutional Networks
- Broadcasting Convolutional Network for Visual Relational Reasoning
- Statistically-Motivated Second-Order Pooling
- Compact Generalized Non-local Network
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Bimodal information analysis for emotion recognition
- Automatic detection of parkinson's disease from words uttered in three different languages
- Predicting children's reading ability using evaluator-informed features
- Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling
- Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining