Vision transformers cut images into a grid of small patches ("tokens") and compare every token with every other one, so cost grows with the square of the token count. Satellite and aerial scenes can be tens of thousands of pixels across. To process them at all, current systems merge tokens that look alike into a single averaged token. This works when the important things look different from their surroundings. It works badly when they do not: a vehicle under netting, a structure painted to match the terrain, unripe fruit among leaves. A camouflaged target is close to its background in exactly the feature space that merging rules use to decide what is redundant. So the compression step itself may delete the target before the deeper layers that could have found it ever see it.
We propose to measure this failure directly and to fix it with a small learned "merge veto." The veto is a lightweight predictor that reads early-layer features together with simple frequency-band statistics. It flags tokens that the uncompressed model will later depend on, even when those tokens look redundant by cosine similarity. The bet is concrete. Camouflaged targets become separable from background deeper in the network than salient targets do, so merging at the usual middle layers erases them. Cheap early cues such as high-frequency texture energy are enough to predict which tokens to spare, at a fixed token budget.
Core Claim
Similarity-driven token merging loses camouflaged targets much more often than conspicuous ones, and the loss grows as target–background contrast falls. This happens even in cases where the full, uncompressed model segments the target correctly. A veto head trained to imitate the uncompressed model's own sensitivity to each merge can recover most of that loss at the same compute. Similarity-only rules cannot, including those designed to protect small objects.
The proposal contributes three things in decreasing order of certainty. The first is a contrast-stratified diagnostic of target survival under merging. The second is the claim that the separation between camouflaged targets and their background emerges later in the network. The third is the veto mechanism. If the diagnostic shows no extra loss for camouflaged targets, the remaining parts are unnecessary and we will say so.
Relation to Prior Work
Two token-reduction methods supply the merging machinery we build on, and both motivate the gap. Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning groups neighbouring tokens with a superpixel-style local k-means and later restores full resolution by nearest-neighbour interpolation. It keeps 99.5% of ADE20K mIoU while saving 30% of GFLOPs, but it notes that clustering cannot happen in early layers because semantics have not formed yet. That observation is the seed of our depth hypothesis. Accelerating Transformers with Spectrum-Preserving Token Merging (PITOME) gives each token an "energy" score from its similarity to neighbours and protects low-energy, isolated tokens from merging. It reports a 0.5% ImageNet drop at 40–60% FLOP savings.
Both methods define "worth keeping" as "dissimilar to neighbours." A camouflaged target is, by construction, similar to its neighbours. Our veto replaces that definition with "the full model's output depends on it." Dissimilarity is not used as the criterion.
In remote sensing, UHR-BAT is the closest existing work. It scores tokens with text-query cross-attention, clusters the image into regions with fast k-means, keeps high-importance tokens, and mean-pools the rest per region. It reports 44.0% weighted accuracy on XLRS-Bench against 43.3% for GeoLLaVA-8K. Its importance signal is attention, which is uninformative when the target blends in, and its regions come from the same RGB-feature clustering that absorbs camouflaged targets. Our work differs in two ways: importance comes from a distilled sensitivity signal rather than attention, and we evaluate explicitly on low-contrast strata.
FIRM addresses a related loss: the boundary detail lost when a coarse token spans several objects, which it handles with sub-token mask codes. That is complementary. FIRM fixes what happens inside a surviving token, while we ask whether the token survives at all.
The camouflage literature constrains and supplies the method. Background matching drives local edge gradients toward zero (Fast Camouflaged Object Detection via Edge-based Reversible Re-calibration Network), which explains why spatial similarity fails. Frequency-Guided Spatial Adaptation for Camouflaged Object Detection (FGSA-Net) shows that concentric Fourier frequency rings expose texture differences invisible in RGB. It uses all tokens and does no compression; we borrow its ring statistics only as cheap input features for the veto. CamoFormer separates foreground and background attention heads in the decoder, reaching weighted F-measure 0.786 on COD10K; it serves as our uncompressed reference for segmentation quality.
Is There Really a Camouflaged Object? is a constraint we must respect. Camouflage detectors fire on empty backgrounds, with false-positive rates up to 97.4% for RUN and USCNet. Any rule that spares "suspicious" tokens risks inflating budgets and hallucinations on the empty terrain that dominates satellite scenes.
Research Program
Diagnostic. We first build a diagnostic of target survival. We insert ToMe-style bipartite merging, PITOME, local k-means clustering, and UHR-BAT-style region preserve-and-merge into a frozen ViT-L segmentation backbone. Merging is applied at the depths and ratios each paper uses. For every ground-truth target, we record two quantities: the fraction of its tokens that end up in clusters whose majority is background, and the final mask IoU.
To stratify by camouflage, we define target contrast as the cosine distance between the target's mean backbone feature and the mean feature of a surrounding ring of equal area. This lets us plot survival against contrast. We do this on natural camouflage data (COD10K, NC4K, PlantCamo) and on overhead data (DIOR, DOTA, DIOCblueberry). On the overhead sets, the lowest-contrast decile acts as a naturally camouflaged subset. We also track a separability probe: a linear probe for target versus background trained per layer. This tests whether camouflaged targets become separable later than salient ones.
Veto head. We then train the veto head. Let $x_i$ be token $i$ at the merging layer, $c(i)$ the centroid it would be merged into, and $g_i$ the gradient of the uncompressed model's task loss with respect to $x_i$. The first-order change in loss caused by merging token $i$ is
$$ s_i = \left| g_i^\top \big(c(i) - x_i\big) \right| , $$
which comes from a Taylor expansion of the loss around the unmerged state. Computing $s_i$ needs one backward pass of the frozen teacher per training image, and it is never computed at inference.
The veto head is a two-layer network, under 1% of backbone parameters. It takes as input $x_i$, the token's FGSA-style ring energies computed on its pixel patch, and the mean feature of its 3×3 neighbourhood. It is trained with a ranking loss to order tokens by $s_i$. At inference, under a fixed budget, the top-$m$ vetoed tokens are exempted from merging and everything else is merged by the host method. The host method therefore keeps its speed, and the veto only reallocates a small protected set.
The training labels also come from images without targets (the empty-background scenes of OPC16K, and target-free DIOR tiles). There $s_i$ is small everywhere, which teaches the head not to spend protection on empty terrain.
Baselines
The naive reference point is running the backbone with no compression on the same tiles. This is the accuracy ceiling and the cost floor we want to approach. Its counterpart is uniform downsampling to the compressed token count, which UHR-BAT shows erases pixel-scale targets.
The strongest small-object-preserving merger is PITOME (Accelerating Transformers with Spectrum-Preserving Token Merging). It is the most direct competitor because it explicitly protects isolated informative tokens, so the question is whether that protection already covers camouflaged ones. For dense prediction, the strongest is local k-means clustering with reconstruction (Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning). For overhead scenes, it is UHR-BAT's region preserve-and-merge, run with its k-means regions and with attention importance taken from a class-name query.
All methods share the same frozen backbone, the same merging layer and token budget (we sweep 40%, 60%, and 75% FLOP savings), and the same segmentation decoder trained once on uncompressed features. Each method gets the same hyperparameter search budget on a held-out validation split.
The file reports no result for token compression on COD10K, NC4K, PlantCamo, or contrast-stratified DIOR (no reported result found on Lacuna for these). We therefore compare against our own runs of these methods and against reported uncompressed numbers. Those reported numbers are CamoFormer's wF of 0.786 on COD10K and PlantCamo's PCNet $S_\alpha$ of 0.880 (PlantCamo). On overhead VQA, the reference is UHR-BAT's 44.0% on XLRS-Bench.
Beating PITOME means the following at matched FLOPs. On the lowest-contrast quintile, the veto should recover at least half the IoU gap between PITOME and the uncompressed model. On the highest-contrast quintile, it should lose no more than 0.5 points. On OPC16K negatives, the background false-positive rate should not rise above the uncompressed model's rate.
Ablations isolate the key ingredient. One drops the ring-energy inputs, so the veto sees only spatial features. Another replaces the gradient label $s_i$ with a ground-truth-mask label, which tests whether distilling the model's own sensitivity matters beyond "protect the target." A third removes negative images from training, which tests the hallucination constraint.
Why not just…?
Why not just use PITOME's energy score, perhaps computed on frequency features? This is the most serious threat, and we test it directly with a variant whose similarity graph is built on concatenated spatial and ring-energy features, with no learning. If it matches the veto, we adopt it, because it is simpler. Our reason to expect a gap is the following. Energy measures dissimilarity to neighbours, and a background matcher's high-frequency statistics can also match its surroundings. Gradient sensitivity can instead pick up the dependence that the deep layers develop through global context.
Why not just fine-tune the backbone with merging switched on, so it learns features that survive merging? This could work, so we run merge-aware full fine-tuning on the same data as a baseline. One complication is that How to Efficiently Adapt Foundation Models to Remote Sensing for Object Detection? found low-shot PEFT fails on DIOR (zero-shot GLIP at 0.04 mAP), with full tuning needed to reach 0.75 mAP. That makes fine-tuning expensive and data-hungry for the rare camouflaged overhead targets we care about. We compare at 10-shot and full-data regimes. A frozen backbone with a tiny veto head should win in the low-shot regime if it wins anywhere.
Why not just tile the scene and skip compression? Tiling has a clear accuracy ceiling, and we include it, but it pays the full quadratic cost per tile. It also cuts the global texture context camouflage detection relies on, as argued by Uncertainty-Guided Transformer Reasoning for Camouflaged Object Detection. Tile seams also cause boundary errors, which in Benchmarking Dense and Indiscernible Object Counting with Blueberries required overlapped crop-and-stitch inference to reduce MAE from 6.36 to 3.63. We report accuracy against wall-clock time so the trade-off is visible rather than asserted.
Proposed Experiments and Decisive Checks
The first check is headroom. On the low-contrast strata, the uncompressed model must segment targets substantially better than the compressed baselines. If it already fails there, compression is not the bottleneck, and the proposal reduces to a negative diagnostic. That negative result is still worth reporting, given that UHR-BAT and PITOME are being adopted for satellite use.
The second check is the depth hypothesis: the layer at which the linear probe separates target from background should be later for low-contrast targets. If separation happens early, plain similarity merging should be adequate and the veto unnecessary.
The third check is the main comparison. We plot IoU-versus-contrast curves for every merger at three budgets on COD10K, NC4K, PlantCamo, and contrast-stratified DIOR and DOTA, and count error on DIOCblueberry. This is paired with background false-positive rate and accuracy on OPC16K, where OPCNet reports FPR_bg 0.050.
The fourth check is that general overhead understanding does not suffer. We run the veto inside a UHR-BAT-style pipeline on XLRS-Bench at matched latency, against UHR-BAT's reported 44.0%.
As a final stress test, we evaluate on DOTA targets perturbed by CamoNet, whose adversarial camouflage drives detector AP from 69.3% to 0.6%. The question is whether merging compounds that attack. We treat this as secondary, since adversarial perturbation differs from physical camouflage.
References
- Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning
- Accelerating Transformers with Spectrum-Preserving Token Merging
- UHR-BAT: Budget-Aware Token Compression Vision-Language model for Ultra-High-Resolution Remote Sensing
- FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentation
- Fast Camouflaged Object Detection via Edge-based Reversible Re-calibration Network
- Frequency-Guided Spatial Adaptation for Camouflaged Object Detection
- CamoFormer: Masked Separable Attention for Camouflaged Object Detection
- Is There Really a Camouflaged Object? Towards Realistic Camouflaged Object Detection
- PlantCamo: Plant Camouflage Detection
- How to Efficiently Adapt Foundation Models to Remote Sensing for Object Detection?
- Uncertainty-Guided Transformer Reasoning for Camouflaged Object Detection
- Benchmarking Dense and Indiscernible Object Counting with Blueberries
- CamoNet: A Target Camouflage Network for Remote Sensing Images Based on Adversarial Attack