Context-Aware Downsampling for Tiny Objects

Most vision networks shrink their feature maps by half several times. An object 8 pixels wide is reduced to a single cell after three such steps, and its signal can be averaged into the surrounding background and lost for good. Learned pooling tries to prevent this by letting a small network decide which pixels matter before each reduction. That network, however, only sees a few neighboring pixels. A lone bright pixel could be a distant swimmer or one more wave crest, and a 3×3 view cannot distinguish the two. The answer depends on what the rest of the row, column, or scene looks like.

We propose giving the pooling decision image-wide context cheaply. Before each downsampling step, the pixels of a feature map are rearranged so that pixels far apart in the image become neighbors in memory. Plain per-pixel linear layers can then compare a candidate pixel against distant background statistics. The rearrangement is a memory operation with no multiply-adds, so total cost stays linear in image size. The resulting importance weights drive a learned weighted average over each 2×2 block. Our claim is that the tiny objects which survive downsampling are the ones a pooling layer can tell apart from their background. Making that judgment non-local should help most on the smallest, most cluttered targets. It should also outperform simpler ways of adding context, such as one global summary vector or a larger convolution.

Core Claim and What Is New

Two lines of work meet here without having been combined. The first shows that downsampling is the main cause of tiny-object loss. Rethinking the backbone architecture for tiny object detection delays downsampling ("bottom-heavy" backbones) and reports a +3.51% mAP gain on TinyPerson. It argues that fixes placed after the backbone come too late. LIP: Local Importance-Based Pooling replaces fixed pooling with weights predicted by a small convolutional "logit module". It raises COCO small-object AP from 21.9 to 24.0 with Faster R-CNN. DPNet instead chooses one downsampling factor per image. These papers motivate the proposal. LIP's logit module is purely local, which the research file flags as a known weakness: its importance maps become noisy when early layers cannot see enough context.

The second line shows that rearranging tensors gives wide context almost for free. AS-MLP and S²-MLP shift channel groups spatially so that 1×1 layers can mix neighbors. Hire-MLP rearranges tokens within and across regions. MixNet permutes spatial and channel axes for global modeling of 4K images. Hi-IR groups dispersed pixels from different patches into new windows. These papers supply the mechanism.

None of them puts the rearrangement inside the downsampling operator. Each one still downsamples with fixed patch merging or a fixed initial reduction; MixNet explicitly downsamples first and cannot recover what that step erases.

The closest combined system is Dome-DETR. Its Axis Permuted Encoder bridges context between sparse foreground windows in the detection encoder, and it reaches 34.6 AP on AI-TOD-v2. Dome-DETR leaves its backbone's downsampling untouched. Our operator sits in the backbone, before information is discarded, and is complementary to it.

The Operator

At each stride-2 step, an input map $X$ of size $C \times H \times W$ is reduced to $C' \times H/2 \times W/2$. We keep LIP's weighted-average form:

$$ Y_{p} = \frac{\sum_{q \in B(p)} \exp\big(g(X)_q\big) \odot X_q}{\sum_{q \in B(p)} \exp\big(g(X)_q\big)}, $$

The symbols are as follows. $B(p)$ is the 2×2 block feeding output cell $p$. $X_q$ is the feature vector at input pixel $q$, and $\odot$ is per-channel multiplication. $g$ is the network that produces per-pixel, per-channel importance logits.

Our change is entirely in $g$. In LIP, $g$ is a few convolutions. Here, $g$ adds a local 3×3 depthwise branch to a context branch. The context branch uses dilated grouping in the style of Hi-IR, applied separately along rows and along columns. Every $s$-th pixel along an axis is gathered into a window of fixed length $k$, and a shared linear layer mixes each window. The pixels it mixes are therefore spread across the whole row or column, regardless of image size.

We choose fixed-length dilated windows over MixNet's full-axis permutation for a practical reason. A linear layer over an entire axis ties its weights to one input width, which breaks under the variable input sizes used in detection. Fixed windows avoid that dependence.

The design targets a total overhead of under 10% of backbone FLOPs. We will also report measured GPU latency, because Hire-MLP notes that non-contiguous memory shuffles can cost real time even at zero FLOPs.

Research Program

We will build the operator as a drop-in replacement for every stride-2 reduction in three backbones. The first is ResNet-50, to match LIP's setting. The second is the bottom-heavy BH-ResNet50, to test whether context-aware pooling adds to delayed downsampling. The third is an AS-MLP-T-style shifted-MLP backbone. The research file notes that such backbones inherit the same aggressive downsampling the bottom-heavy paper criticizes.

The analysis centers on survival probing. For each frozen backbone and each stride (8, 16, 32), we train a linear probe to predict whether a ground-truth tiny object is centered in a given feature cell. We stratify by object size and by background clutter, measured as local image variance around the object. If the claim holds, our operator will keep tiny objects linearly detectable at deeper strides than LIP does. The gap should widen as objects shrink and backgrounds get busier.

Baselines

The naive reference is the unchanged backbone with its standard strided convolution or patch merging. It is trained with the same detector, schedule, augmentation, and input resolution as every other arm. All gains are measured against it.

The strongest existing downsampling competitor is LIP, run in the same backbones. We give its logit module enough extra channels to match our operator's parameter count and FLOPs, so that any gain from context is not really a gain from capacity. On COCO with Faster R-CNN and ResNet-50, LIP reports $AP_s$ of 24.0 versus 21.9 for the baseline (LIP). Our first bar is a reproduced LIP number, with $AP_s$ above it by more than seed variance; we will use three seeds.

The strongest backbone-level competitor is BH-ResNet50 (Rethinking the backbone architecture). Its reported gain is +3.51% mAP on TinyPerson over ResNet-50. The file gives no absolute value, so we reproduce both. Beating it means either higher TinyPerson mAP at matched FLOPs or a further gain when our operator is added inside BH-ResNet50.

The strongest full detector on AI-TOD-v2 is Dome-DETR-L at 34.6 AP, against 31.3 for its D-FINE baseline (Dome-DETR). As a final test, we swap our operator into Dome-DETR's backbone downsampling and keep the rest of its training recipe fixed. We count it a win if AP and especially very-tiny AP rise above a reproduced Dome-DETR, with GFLOPs inside its reported 338.4–398.9 range.

Two ablations isolate the key ingredient. In the first, the context branch is replaced by a squeeze-and-excitation-style global-average vector. That keeps global context but removes spatial structure along rows and columns. In the second, the dilated grouping is replaced by an ordinary contiguous window of the same length, which keeps capacity but removes reach. Both use the same parameter count and tuning budget.

Why not just…?

Why not just downsample losslessly, by moving each 2×2 block into channels and mixing with a 1×1 layer? This is a pure permutation, much like Hire-MLP's inner-region rearrangement, and it discards nothing at the reduction step itself. It is a serious threat, so we include it as an arm at matched FLOPs. Our expectation is that the loss simply moves to the next channel-mixing layer: without a judgment about which pixels matter, a tiny object's channels are still blended with background. The survival probes test this directly. If this lossless arm matches ours, we will report that importance weighting is unnecessary and that losslessness was the active ingredient.

Why not just give LIP's logit module a bigger receptive field with dilated convolutions or a global summary vector? The global-vector ablation covers the summary option. A dilated-convolution logit module at matched FLOPs will be run as a further arm. Our argument is that clutter such as sea waves or road texture is organized along lines and over long ranges. A single global vector cannot localize it, and dilated convolutions cover it only sparsely at a fixed cost per pixel. If either alternative matches our operator on TinyPerson and AI-TOD-v2, the claim that row and column grouping matters is false, and we will say so.

Why not just upscale the input, or adopt a bottom-heavy backbone, and skip new operators? DPNet reports that upscaling inflates compute and adds background negatives (DPNet). We still run a 1.5× input-upscaling arm at roughly matched GFLOPs. Bottom-heavy design is complementary, not competing, which is why BH-ResNet50 is one of our host backbones. The real question is whether our operator adds gains on top of it.

Proposed Experiments

Training and evaluation use TinyPerson, AI-TOD-v2, VisDrone, and COCO 2017. On TinyPerson we report $AP_{50}$ across tiny-size bins. On AI-TOD-v2 we report AP with size breakdowns, and on COCO we report $AP_s$. For TinyPerson we pretrain on COCO, following the setting of Scale Match. ImageNet-1K Top-1 is the guard against the large-object accuracy loss seen in bottom-heavy backbones. For every arm we also report GFLOPs, parameter count, and measured throughput on one GPU.

The first decisive check is whether our operator beats parameter-matched LIP and the lossless space-to-channel arm on the smallest size bins. The second is whether that margin grows with background clutter, as the survival probes predict. Either outcome is informative: a positive result places context-aware pooling in the backbone, and a null result rules it out.