Keeping the High Frequencies Where Small Objects Live: Residual-Band Visual Tokens for Aerial Vision-Language Models

Vision-language models turn an image into a sequence of "visual tokens" that a language model reads alongside text. High-resolution aerial images produce tens of thousands of these tokens, so practical systems compress them. A recent and cheap way to compress is to treat the grid of token features like an image and low-pass filter it: take a discrete cosine transform (DCT, which re-expresses a grid as a sum of smooth-to-sharp cosine patterns), keep only the smooth coefficients, and transform back to a smaller grid. This works well on everyday photos. It is close to the worst possible choice for overhead imagery, though, where a runway spanning thousands of pixels sits next to cars only a few pixels wide. The cars are exactly the sharp, local detail that low-pass filtering deletes.

We propose to spend a fixed token budget in two parts. The first part is a small low-pass grid that carries large structures and scene layout. The second is a sparse set of "residual-band" tokens, each encoding only what the low-pass grid failed to reconstruct, placed at the few locations where that reconstruction error is largest. In two sentences: compression should discard high frequencies everywhere except where they carry objects, and the reconstruction error of the low-pass grid is a free, task-agnostic map of where those places are. If this holds, a single forward pass should recover much of the small-object accuracy that current aerial models obtain only through slow multi-pass zooming.

Core Claim

At an equal number of visual tokens, a model reading a low-pass grid plus top-k residual tokens will beat uniform low-pass compression on small objects in aerial scenes and will not lose accuracy on large ones. The gain will come from two separable ingredients. The first is the selection signal: where the residual is large. The second is the content: tokens that carry the high-frequency residual rather than duplicating the raw feature. We will test each ingredient against a purely spatial version of itself, because if spatial versions match, the frequency view is unnecessary.

Where This Sits

The mechanism we start from is Fourier-VLM. It inserts a parameter-free DCT low-pass compressor between the vision encoder and the language model, cutting FLOPs by up to 83.8%. It loses only 3.87% performance when keeping 36 of 576 tokens, but it applies one global cutoff, and it reports blurring and hallucination on small objects and dense text. We keep its low band unchanged and add the residual band. Fourier-VLM was evaluated only on natural-image and video benchmarks, so its failure mode has not been measured where it should hurt most.

Spatial projectors pursue the same token-budget goal without frequencies. Honeybee preserves locality with convolutions or deformable attention before pooling. TokenPacker injects multi-layer detail into coarse point queries via cross-attention, but degrades sharply below 32 tokens. LLaVA-UHD v2 builds a high-resolution pyramid with window attention. All three spread their budget uniformly over the image grid. Our proposal instead makes the allocation content-dependent and nearly free to compute.

The application regime, and the latency target, come from RSCoVLM. It handles extreme scale variation with a "zoom-in chain": a coarse pass proposes regions, which are then cropped and re-encoded. This yields a 35% improvement on ultra-high-resolution benchmarks, at the cost of sequential passes. Our residual map plays the role of the coarse pass inside a single encoding.

The closest prior use of frequency decomposition in aerial vision-language work is FVMGN. It uses wavelet sub-bands and text priors for cross-sensor scene classification, reaching 93.19% overall accuracy on MUUFL→Houston 2013. It does no token compression and produces no boxes or answers, so it supports the value of band separation but not our mechanism.

Two lines of work act as constraints rather than motivation. Deep nested U-structure network with frequency attention shows that DCT-based attention confuses spectrally similar non-targets and shadows. The file's summary of spectral methods likewise notes that high-frequency content is contaminated by sensor noise and speckle. Both predict that raw residual energy will also fire on forests, wave fields, and roof textures, and we design around that below.

We deliberately do not pursue the amplitude/phase style-transfer idea of FDA and Phase Consistent Ecological Domain Adaptation inside the projector. Cross-sensor robustness is a separate claim with separate confounds. Folding it in would make neither result interpretable.

Research Program

The encoder runs at native dynamic resolution, as in Qwen2.5-VL, so the token grid before compression is large enough that a few-pixel vehicle still perturbs some token features. Compression acts on this grid.

Let $X$ be the encoder's token grid. Let $L(X)$ be the DCT low-pass reconstruction at cutoff $c$, upsampled back to full grid size. The residual is $R = X - L(X)$. The low band contributes the $m$ tokens of the truncated grid, exactly as in Fourier-VLM. We tile $R$ into small windows $w$ and score each window by its residual energy:

$$s_w = \frac{\lVert R_w \rVert^2}{\operatorname{median}_{w' \in \mathcal{N}(w)} \lVert R_{w'} \rVert^2}$$

Here $\mathcal{N}(w)$ is a neighborhood of surrounding windows. The ratio rewards residual energy that stands out from its surroundings, which is meant to suppress uniformly textured regions such as forests and sea clutter. The top $k$ windows are each pooled into one token by a small shared projection of $R_w$. Each such token carries a 2D position embedding and a band-indicator embedding, so the language model knows it is reading a correction to a coarse token rather than a standalone patch.

The total budget is $m + k$, matched to every baseline. We study the split between $m$ and $k$ as a function of total budget. We also study one extension with three bands at two cutoffs, a Laplacian-pyramid-like arrangement that tests whether medium-sized objects benefit from an intermediate band.

Training follows the two-stage recipe of the base model: projector-only alignment, then instruction tuning. We follow the evidence in Scalable Vision Language Model Training via High Quality Data Curation that isolating projector training early stabilizes the bridge. Instruction tuning uses the remote-sensing SFT data released with RSCoVLM, which covers detection with quadrilateral text coordinates, grounding, and question answering.

The analysis side asks three questions. Does residual-energy selection actually concentrate on ground-truth small objects? We measure this as recall of DOTA object centers within selected windows. Does the normalized score avoid texture regions? Does the learned optimum of $m$ versus $k$ shift with image scale and object density in a predictable way?

Baselines

The naive reference is the same base model and training run with no compression at the largest token count memory allows. It gives an upper reference for accuracy and a lower reference for speed. Alongside it we run plain bilinear pooling to the matched budget, the simplest thing anyone would do.

The most direct competitor is uniform DCT low-pass compression as in Fourier-VLM, using all $m + k$ tokens in the low band. It is the strongest competitor because our method is literally this plus a reallocation, so any gain is attributable to the residual band. Its reported figure is a 3.87% drop at 36 tokens relative to the uncompressed LLaVA-1.5-7B. We found no reported result on Lacuna for Fourier-VLM on any remote-sensing benchmark, and we will produce that number. Beating it means higher small-object AP on DOTA and higher counting and presence accuracy on aerial VQA at every matched budget from 144 down to 36 tokens, with large-object AP within noise.

The strongest spatial projectors are TokenPacker, which reports 70.6% on TextVQA, and Honeybee's C-Abstractor, whose 13B model reports 73.5 on MMBench. Both are reimplemented on the same base model, data, steps, and token budgets. Each gets the same hyperparameter-search budget as our cutoff and window-size search.

The strongest aerial system is RSCoVLM with its zoom-in chain. On DOTA-v1.0 it reports 55.50 $AP_{nc50}$, and on ultra-high-resolution benchmarks a 35% gain from zooming. We report both its accuracy and its wall-clock latency on identical hardware. For us, beating it means matching most of its accuracy in one pass at a fraction of its time, not exceeding it outright.

Two ablations isolate the key ingredients. The content ablation keeps our selection but feeds the raw token features $X_w$ instead of residuals $R_w$. The selection ablation keeps residual content but chooses windows by encoder attention or feature norm, or by a learned scorer, instead of residual energy. A third ablation removes the neighborhood normalization.

Why not just…?

Why not just prune tokens spatially, keeping the most salient ones? This is the most serious threat. A spatial method that keeps a pooled grid plus the top-k original tokens by local variance is nearly our method in different clothing. Our content and selection ablations are exactly this alternative. If raw-token selection matches residual tokens at equal budget, we will conclude that the frequency view adds nothing beyond a convenient saliency score, and we will report it as such. Our expectation that residuals win rests on redundancy: a raw token re-sends the low-frequency information already in the coarse grid, while a residual token spends its capacity only on what is missing. This should matter most at tight budgets like 36–64 tokens, where TokenPacker already collapses.

Why not just use a locality-preserving spatial projector like Honeybee or TokenPacker? They allocate tokens uniformly, so a scene that is 99% farmland spends 99% of its budget on farmland. Honeybee's own 7B variants still hallucinate objects, and TokenPacker degrades below 32 tokens. Neither was tested on overhead scenes. The matched-budget comparison on scale-stratified DOTA AP will settle this. We also report the Pearson correlation between object size and accuracy, the scale-fairness measure used for PKINet.

Why not just zoom in, as RSCoVLM does? Zooming is the accuracy ceiling and may remain better on the hardest ultra-high-resolution questions. Its cost is extra sequential encoder and language-model passes. We present the comparison as an accuracy–latency curve, not a single number. If one-pass residual tokens recover only a small share of the zoom gain, the method's value reduces to efficiency on easy scenes, and we will say so.

Why not just use less aggressive uniform compression? If 144 uniform DCT tokens already match full resolution on aerial benchmarks, there is no problem to solve. We therefore sweep the uniform baseline across budgets first. The proposal is only worth continuing if a clear small-object gap appears at practical budgets.

Proposed Experiments

The main testbed is generative oriented detection on DOTA-v1.0, scored by $AP_{nc50}$ as in RSCoVLM and broken down by object-area bins. Alongside it we run RSVQA-HR, where the original RSVQA baseline reports 83.23% overall accuracy and counting is the weakest category. We also run LRS-VQA-Zoom and VRSBench for ultra-high-resolution reasoning. TextVQA and MMBench check that the method does not hurt general and text-dense images; dense text is the other place where Fourier-VLM reports failure, so TextVQA is a secondary win condition. All runs share base model, data, steps, and budgets (144, 64, 36), with three seeds for the compressed variants.

The decisive checks are four. First, small-object AP must rise over uniform low-pass at matched budget, while large-object AP stays flat. Second, residual tokens must beat raw-token selection, and residual-energy selection must beat attention-based selection; otherwise the frequency claim fails. Third, the normalized score must cut the share of selected windows falling on vegetation and water relative to the unnormalized score, directly testing the noise and texture concern raised by the frequency-attention segmentation work. Fourth, the single-pass method must sit strictly above the uniform baseline on the accuracy–latency plane against RSCoVLM's zoom chain. If the first two checks fail, the idea is wrong. If only the fourth fails, it survives as an efficiency trick rather than a replacement for zooming.