A Vision Transformer cuts an image into patches and then compares every patch with every other one. The cost of those comparisons grows with the square of the patch count, so doubling image resolution makes attention about sixteen times more expensive. A popular way to cut this cost is to spend patches unevenly. Adaptive Patch Transformers measure how "busy" each region is using local pixel entropy, which is roughly the unpredictability of its intensity histogram. Flat regions get large, cheap patches and detailed regions get small, expensive ones Accelerating Vision Transformers with Adaptive Patch Sizes. This works on ImageNet, COCO and ADE20K. But entropy cannot distinguish structure from sensor noise. In particle-velocimetry photographs, speckled synthetic-aperture radar, low-light captures and infrared thermography, the meaningless background is often the highest-entropy part of the frame. An entropy-driven allocator would then spend its finest patches on noise and give coarse patches to the faint defect or particle that matters.
We propose to measure this failure and then test a cheap fix. The measurement is a cross-domain audit of where entropy-driven and attention-driven token budgets actually land. The fix replaces raw entropy with excess entropy above the local noise floor, an allocation score that drops back to plain entropy on clean images. The broader point concerns evaluation scope. Every token-budget heuristic in this literature was calibrated on consumer photographs. We argue that the efficiency claims of quadratic-attention workarounds cannot be trusted until they are checked on the imaging regimes where they are most likely to be deployed and most likely to break.
Core Claim
The claim is falsifiable. Raw-entropy patch allocation degrades in a predictable, dose-dependent way as sensor noise rises. Most of that loss can be recovered at near-zero overhead by subtracting an estimate of the entropy that noise alone would produce in each region.
Concretely, let $H(r)$ be the multi-scale histogram entropy that APT already computes for region $r$. Let $\hat\sigma_r$ be a local noise level estimated from the finest wavelet detail band. Let $H_0(\hat\sigma_r)$ be the entropy of a pure-noise patch at that level, computed once by lookup table. The proposed allocation score is
$$ s(r) = \max\big(0,\; H(r) - H_0(\hat\sigma_r)\big), $$
and it is thresholded exactly as APT thresholds $H(r)$. On a clean image $\hat\sigma_r \approx 0$, so $s(r)$ equals APT's own score and nothing changes.
The wavelet route comes from the particle-velocimetry analysis in Information Entropy Analysis of a PIV Image Based on Wavelet Decomposition and Reconstruction. That study separates particle signal from heavy background noise by wavelet decomposition and reports that entropy shifts sharply once the noise is removed: from 5.65 to 7.53 bits/pixel in the separation-flow case. We reuse its decomposition as a local noise estimator rather than as a restoration step.
Relation to Prior Work
Three groups of papers play different roles here.
The motivating evidence comes from two places. APT and AgilePruner both flag their thresholds as calibrated on standard data. AgilePruner, which sets a visual-token pruning threshold from a model's effective rank and attention entropy, documents over-pruning of small clustered objects in otherwise simple images AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models. Separately, An Improved SPIHT Algorithm for Image Compression in Low Bit Rate shows natural-image assumptions failing on noisy and high-contrast content.
The application regimes come from three papers. Solar thermography requires isolating small hotspots and snail trails Detection and analysis of deteriorated areas in solar PV modules using unsupervised sensing algorithms and 3D augmented reality. Low-light images are shown to defeat models trained under normal illumination Retinexmamba: Retinex-Based Mamba for Low-Light Image Enhancement. Speckled radar change detection is where ChangeRWKV, a linear-time recurrent model, was tested Beyond Quadratic: Linear-Time Change Detection with RWKV.
The constraints come from two further papers. First, Exploring Token Pruning in Vision State Space Models shows that deleting tokens from scan-based models costs more than 68% accuracy unless the scan state is realigned. We therefore restrict variable patching to attention models and treat linear recurrent models only as full-grid competitors. Second, the pansharpening study Cross-Scale Pansharpening via ScaleFormer and the PanScale Benchmark is the closest existing work linking narrow evaluation to quadratic cost. It shows that training on 256×256 crops hides out-of-memory collapse and tiling seams at 2000×2000. It processes every token uniformly, however, and never asks whether content-adaptive budgets survive a domain shift. That question is ours.
Research Program
The program has three parts.
The first part is an allocation audit, and it needs no training. We run APT's entropy allocator, SG-Former's learned significance map SG-Former: Self-guided Transformer with Evolving Token Reallocation, and our corrected score on images that have a known foreground mask:
- particle regions in the PIV flow data;
- annotated defect boxes in Solar Panel Infrared Images v5 and Cali-Thermal;
- change masks in LEVIR-CD and SAR-CD;
- ADE20K with additive Gaussian and Poisson noise at controlled levels.
The audit statistic is the fraction of fine patches that fall on the foreground, relative to its area share. It shows directly whether the budget goes to signal or to noise.
The second part is downstream accuracy at matched cost. We fine-tune ViT backbones with each allocator on the following tasks:
- infrared defect detection on Solar Panel Infrared Images v5;
- change detection on LEVIR-CD and SAR-CD;
- enhancement on LOLv2-real, as a dense regression task;
- the synthetic-noise ADE20K sweep.
We report accuracy against token count, FLOPs, and wall-clock time. Wall-clock uses APT's FlashAttention sequence packing, because theoretical FLOP savings often fail to become real speedups.
The third part is a failure analysis. It separates two failure types: noise inflation, which the corrected score targets, and faint small targets inside genuinely flat regions, which it does not obviously fix. We report the second honestly rather than claiming it.
Baselines
The do-nothing reference is a uniform-patch ViT of the same size at the same input resolution. It defines both the accuracy ceiling for this backbone and the cost that adaptive methods try to cut. A second naive point is uniform downsampling to the same token count as each adaptive method. It tells us whether adaptivity is doing anything beyond reducing resolution.
The strongest direct competitor is APT with raw entropy Accelerating Vision Transformers with Adaptive Patch Sizes. It is the method whose heuristic we claim breaks. It reports a 30% input-token reduction on COCO and a 28–32% reduction on ADE20K without degradation, and up to 40–50% speedups for ViT-L and ViT-H. We use its released thresholds as the default, and we also retune its threshold on each target domain's validation split with the same search budget we give our method. That way any gain cannot be credited to a threshold-calibration mismatch.
For change detection, the strongest full-grid efficient model is ChangeRWKV-B. On LEVIR-CD it reports 85.46% IoU and 92.16% F1 at 134.25G FLOPs and 6.5GB memory for a 1024×1024 input. We re-run it with our training schedule and report both its published figures and our reproduced figures. For SAR-CD, Solar Panel Infrared Images v5 detection, and PIV there is no reported result found on Lacuna for a comparable learned model, so our uniform-patch baseline sets the reference there. The Solar PV paper's unsupervised segmentation reaches a Jaccard index of 0.7680, but that is a different task and serves only as context.
SG-Former's learned significance map is the learned-allocation competitor. It uses the same backbone scale and training epochs.
The key ablation swaps only the allocation score between raw entropy, excess entropy, and a denoise-then-entropy variant, holding the patch sizes, the aggregation MLPs and the token budget fixed. Beating a baseline means one of two things at an equal token budget. Either the method recovers at least half of the accuracy gap between raw-entropy APT and uniform patching on the noisy domains while staying within noise of APT on clean ADE20K, or it matches the accuracy while cutting tokens by 25% relative to raw-entropy APT.
Why not just…?
Why not just denoise the image before computing entropy? This is the most serious threat, and we test it directly as the denoise-then-entropy ablation. A pre-denoiser, such as the wavelet reconstruction of the PIV study, costs a full image-to-image pass. It can also erase exactly the faint snail trails and hotspots the task needs, whereas our score only changes where patches go and never alters the pixels the network sees. If denoise-then-entropy matches our score in both accuracy and wall-clock time, the contribution shrinks to the audit and the finding that raw entropy must never be used unfiltered. We would report it that way.
Why not just learn the allocation end-to-end? SG-Former learns a significance map from attention, and learned routers can in principle learn to ignore noise. But they learn from whatever domain they are trained on. The standard-trained versions are the ones deployed as plug-ins, and AgilePruner shows that attention-derived importance fails in its own characteristic ways. Our experiments compare SG-Former both off-the-shelf and fine-tuned per domain. If fine-tuned SG-Former dominates on every domain, a training-free score has value only when no in-domain labels exist, and we will say so.
Why not just skip allocation and use a linear-time full-grid model? Models like ChangeRWKV or QT-ViT, a second-order Taylor linear-attention ViT QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion, avoid the quadratic cost without any heuristic. This is a fair alternative, and the matched-cost LEVIR-CD and SAR-CD comparison settles it on the axis we care about: accuracy per wall-clock second. Our proposal survives only if corrected-entropy ViTs sit on or beyond that Pareto curve. The two approaches are also complementary, but ToP-ViM's results suggest that variable token sets do not transfer to scan models without extra machinery.
Why not just tile the image? The ScaleFormer study shows that tiling produces seam artifacts and context loss at full scale. Tiling also leaves the per-tile allocation problem untouched, so it is not an alternative to the question asked here.
Decisive Checks
The first decisive check is the synthetic-noise ADE20K sweep. If raw-entropy APT's mIoU gap to uniform patching does not widen with noise level, the central premise is false and the project stops at the audit.
The second is the foreground-share statistic on PIV and SAR-CD. If raw entropy already concentrates fine patches on the foreground, noise inflation is not the mechanism, whatever the accuracy results say.
The third is the clean-data control. On unperturbed ADE20K and COCO, the corrected score must stay within seed variance of APT's reported behavior. Otherwise it is a different heuristic rather than a correction.
Finally, all speed claims are reported in measured wall-clock time on the packed-attention kernel, not in FLOPs alone.
References
- Accelerating Vision Transformers with Adaptive Patch Sizes
- Information Entropy Analysis of a PIV Image Based on Wavelet Decomposition and Reconstruction
- AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models
- An Improved SPIHT Algorithm for Image Compression in Low Bit Rate
- Detection and analysis of deteriorated areas in solar PV modules using unsupervised sensing algorithms and 3D augmented reality
- Retinexmamba: Retinex-based Mamba for Low-light Image Enhancement
- Beyond Quadratic: Linear-Time Change Detection with RWKV
- Exploring Token Pruning in Vision State Space Models
- Cross-Scale Pansharpening via ScaleFormer and the PanScale Benchmark
- SG-Former: Self-guided Transformer with Evolving Token Reallocation
- QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion