Shifted Fixed-Kernel Temporal Attention for Tiny-Object Video Detection

Detectors on drones and small robots see video, but most of them look at one frame at a time. For objects smaller than about 12×12 pixels, a single frame carries almost no texture, so the most reliable signal is motion: a small cluster of pixels that moves coherently against the background. The current best way to use that signal in a lightweight detector compares every location in the current frame with many locations in past frames. These comparisons, or the learned sparse sampling that replaces them, are cheap in arithmetic but expensive in memory traffic. On edge hardware, that memory traffic is what sets the frame rate.

We propose to split temporal attention into two parts. Temporal attention here means letting each location in the current frame pull information from related locations in earlier frames. The first part decides where to look: a small, content-dependent displacement estimated on a very coarse grid of image tiles. The second part decides how to weight the neighbourhood around that point, using a fixed Gaussian kernel with no learned query or key weights. Because the displacement is constant within each tile and the kernel is fixed, the whole operation runs as contiguous shifts plus a depthwise convolution, with no scattered memory reads. Combined with a cache of narrow-channel features from past frames, this should give a single-frame YOLO-class detector most of the tiny-object gain of explicit motion matching. It should do so at close to single-frame latency.

Core Claim

Hard-Coded Gaussian Attention for Neural Machine Translation replaced learned self-attention with fixed Gaussians centred on each token. It matched the baseline (30.3 vs 30.0 BLEU on IWSLT16 En-De) and decoded 30.2% faster. The same paper also found that hard-coding cross-attention, where one sequence attends to a different one, cost 5 to 10 BLEU, because the relevant positions move around depending on content. Attending from the current frame to a past frame is a cross-attention problem of exactly this kind. Camera ego-motion and object motion move the relevant location. A naive port of fixed Gaussians to video should therefore fail, and that paper constrains the design.

Our claim is that, for tiny objects in aerial video, the content-dependent part of temporal cross-attention is low-dimensional. It is mostly a per-tile translation. Everything else can be a fixed local kernel. If this holds, the dynamic routing that cross-attention needs can be confined to a handful of coarse offsets per frame, and the dense part becomes input-agnostic and hardware-friendly.

The idea differs from the closest work as follows. XS-VID / YOLOFT builds a multi-scale correlation pyramid, meaning dense matching at every location and scale. DEFA shows that learned sparse deformable sampling spends over 60% of GPU runtime on scattered grid sampling, and fixes this with custom silicon. We aim to remove the scattered access by construction, on commodity hardware.

Research Program

We build on a YOLOv8-S-class detector, the same family YOLOFT extends.

At each pyramid level, the current frame's features are reduced to a narrow width, about 1/8 of the channels. This follows the observation in SlowFast Networks for Video Recognition that a low-capacity temporal pathway costing roughly 20% of compute carries the motion signal. These narrow features go into a ring buffer of the last $T$ frames, so past frames are never recomputed. This streaming reuse is cheaper than the keyframe/non-keyframe split of Progressive Sparse Local Attention and QueryProp, which still run heavy ResNet backbones.

Displacement is estimated once per frame pair at the coarsest level. A small correlation over a ±2-cell window on a tile grid gives one integer-plus-subpixel offset $o_{t,\tau}(g)$ for each tile $g$, between the current frame $t$ and a past frame $\tau$. Each cached map is shifted tile-wise by this offset, which is a contiguous copy. It is then filtered with a fixed Gaussian depthwise kernel. Several widths $\sigma_h$ act as separate heads, as in the hard-coded attention paper. The aligned features are

$$\tilde F_\tau = \sum_h w_h \, G_{\sigma_h} * \mathrm{shift}\big(F_\tau,\, o_{t,\tau}\big),$$

where $G_{\sigma_h}$ is a 2D Gaussian with width $\sigma_h$ and $w_h$ is a learned scalar per head. A per-channel gate then mixes $\tilde F_\tau$ across past frames into the neck. The only learned temporal parameters are the coarse offset head, the gates, and the head scalars.

The analysis has three parts. First, we measure the empirical distribution of displacements of ground-truth tiny objects relative to their tile's offset. This directly tests whether the residual displacement after the coarse shift is small enough for a fixed Gaussian to absorb. Second, we profile memory traffic and kernel time against grid-sample implementations at a matched number of sampled points. Third, we stratify accuracy by camera-motion magnitude and object size.

Baselines

The naive reference point is the single-frame YOLOv8 detector. On XS-VID it reaches 33.6 AP, as reported in XS-VID / YOLOFT. Any temporal module must clear this number at near-identical latency to justify itself.

The strongest existing method is YOLOFT, which reports 36.4 AP on XS-VID for YOLOFT-L and 16 ms per frame for YOLOFT-S. We will train YOLOFT-S and our model from the same YOLOv8-S initialization, with the same schedule, input resolution, number of past frames, and hyperparameter search budget. Both will be timed on the same desktop GPU and a Jetson-class module. Beating YOLOFT means matching YOLOFT-S AP within 0.5 at lower measured latency and DRAM traffic, or exceeding its AP at equal latency. Matching AP alone at equal speed is not a win.

The most direct software competitor to our "where to look" design is a deformable temporal attention neck in the style of TransVOD. We will implement it with DEFA's probability-aware point pruning applied in software and the same number of sampling points. This isolates scattered sampling against tile shifts.

Three ablations isolate the key ingredient. The first removes the coarse offsets: a fixed Gaussian centred at the same location, which is the direct analogue of hard-coded cross-attention and is predicted to fail under ego-motion. The second replaces the fixed Gaussian with a learned depthwise kernel. The third uses per-pixel offsets instead of per-tile offsets.

Why not just…?

Why not just stack past frames or frame differences as extra input channels? This is the most serious threat. For a static camera, frame differencing highlights moving dots almost for free. Drone footage, however, has continuous ego-motion that makes raw differences light up every edge. We will include early-fusion stacking both with and without a single global homography compensation. If globally compensated stacking matches our model on XS-VID, that is a real finding. It would mean per-tile routing is unnecessary, and we would report it as such. The camera-motion stratification is designed to settle this question.

Why not just shrink YOLOFT's correlation pyramid? Reducing the search radius or the number of levels cuts cost, but it keeps dense per-pixel matching at fine scales, where most memory traffic lives. We will run a reduced-radius YOLOFT sweep and plot it on the same accuracy–latency curve. If it dominates our curve, the coarse-offset hypothesis is wrong.

Why not just use deformable attention with pruning? DEFA loses only about 0.8 mAP on COCO, but its 10.1× to 31.9× speedup comes from a dedicated 40 nm accelerator, not from software. Its pruning reduces the number of points while leaving each access scattered. The matched-point profiling comparison will show whether pruned grid sampling on a Jetson closes the gap.

Proposed Experiments

The main benchmark is XS-VID, using AP and per-frame latency. The secondary benchmark is the VisDrone video sequences used by YOLOFT. The file reports only a 19.2% relative gain on extremely small targets there, and no reported absolute result was found on Lacuna for a lightweight temporal baseline on these sequences. We will therefore report all numbers ourselves under one protocol.

ImageNet VID serves as a boundary test. Its objects are large and its motion is non-rigid, so we expect little gain, consistent with YOLOFT's own scale limitation. The strongest efficient reference there is QueryProp at 80.3% mAP and 45.6 FPS with ResNet-50, reported in QueryProp. We do not expect to beat that number with a YOLO-S backbone, and we will report the accuracy–latency trade-off rather than claim superiority.

Four checks are decisive. First, the no-offset ablation should lose most of the temporal gain on high-ego-motion sequences while staying competitive on near-static ones. This would reproduce, in video, the self-attention versus cross-attention split found in translation. Second, the fixed Gaussian should match the learned kernel within noise. If it does not, the input-agnostic claim fails. Third, the residual displacement of objects after the tile shift should mostly fall within $2\sigma$ of the widest head. Fourth, measured DRAM bytes per frame should drop relative to both YOLOFT-S and pruned deformable attention, with latency following DRAM bytes more closely than FLOPs.