A vision-language model such as CLIP (an image encoder trained to match whole images with captions) can be made to label every pixel of an image with an arbitrary text category, without any mask training, by small changes to its final attention layer. These "training-free" tricks work on images of about 448 pixels. They have not been shown to work on 4K street scenes, satellite tiles, or drone mosaics. For those images there are two options today. You can shrink the image, which erases small objects. Or you can slide a window over every tile, which costs compute in proportion to the pixel count and discards the global context each tile needs to know what it is looking at. The strongest training-free system at high resolution avoids the context problem by stacking three foundation models (CLIP, DINO and SAM). That makes memory and compute even heavier.
We propose a single frozen CLIP that runs a cheap pass over the whole image at low resolution and then spends a fixed budget of high-resolution tile passes only where extra resolution changes the answer. Two training-free ingredients drive it. The first is a resolution-sensitivity map: we compare the model's predictions at two cheap scales and zoom where they disagree, rather than where the coarse prediction is merely uncertain. The second is cross-scale attention: inside each zoomed tile, the final attention layer also attends to the low-resolution tokens of the whole image, so the tile inherits global context without SAM or DINO. Peak memory stays at one tile pass plus a few hundred global tokens, whatever the input resolution.
Core Claim
Under a fixed compute budget, a single frozen CLIP with resolution-sensitivity gating and cross-scale attention should reach or exceed the accuracy of exhaustive sliding-window inference while processing only a fraction of the tiles. It should also lose far fewer small objects than coarse-to-fine gating driven by confidence alone. The second half of the claim tests a specific hypothesis raised by combining the two literatures. Coarse-to-fine pipelines fail when the coarse pass misses an object. SparseMat and Chandran's attention-driven cropping both discard whatever their low-resolution stage omits. Vanilla CLIP's final layer is known to smear spatial detail, which makes it a bad coarse stage. We therefore expect the attention fixes from SCLIP and NACLIP to matter more in the coarse pass of a gated pipeline than they do on standard benchmarks. SCLIP replaces query-key attention with self-correlation of tokens and raises CLIP's eight-benchmark average from 14.1% to 38.2% mIoU. NACLIP uses key-key attention with a Gaussian neighbourhood prior and drops the final feed-forward block. Nobody has measured how these fixes affect where a pipeline chooses to look.
Relation to Prior Work
The closest work is Trident. It is training-free and handles high resolution by encoding all sub-images with CLIP, splicing their features into one canvas, and restoring cross-tile coherence with an affinity matrix built from SAM's high-resolution encoder, with DINO guidance and SAM refinement on top. It reports an average gain of 12.0% mIoU over segment-then-splice tiling. We keep its central insight that tiles need global context. We drop the external models and the uniform processing of every tile, and place the context inside CLIP's own final layer.
MROVSeg also uses one CLIP backbone with a global view and local patches. However, its adapter and decoder are trained on COCO masks, and it processes a uniform grid. It is a reference for what supervision buys rather than a direct competitor.
From the memory side, three papers supply mechanisms we adapt, and their shared failure motivates the gate design. BPT shows that spending tokens unevenly, with few on flat regions and many on structure, lowers memory and beats fixed dual-branch networks. It is closed-set and supervised, and its clustering adds runtime overhead. Our tile-level selection is a coarser, GPU-friendly version of the same allocation idea. SparseMat and PatchRefiner V2 both route high-resolution computation using a coarse prediction, and both are tethered to that prediction's errors. This is exactly the failure our resolution-sensitivity score is designed to reduce.
Research Program
The pipeline has three stages, all on frozen CLIP weights with no training.
Coarse passes. The full image is resized and encoded twice, at a low scale $s_1$ (e.g. 224 on the short side) and a medium scale $s_2$ (e.g. 448). Both passes use a NACLIP-style final layer and produce per-patch class probabilities for the text vocabulary. The medium pass is kept as the global context.
Resolution-sensitivity score. For each candidate high-resolution tile, we average the divergence between the $s_1$ and $s_2$ class distributions over the region the tile covers. The intuition is a cheap extrapolation: regions whose labels already change between two coarse scales are the ones where native resolution is most likely to change them again. Regions that are stably labelled, such as sky, road or water, are left to the upsampled coarse prediction. Tiles are selected greedily under a budget $B$, defined as a fraction of the tiles an exhaustive sliding window would process. A small fixed share of the budget goes to stratified random tiles, so that there is always some recall for objects invisible at both coarse scales. We will compare this score against plain entropy of the $s_2$ prediction and against random selection.
Cross-scale attention in zoomed tiles. Each selected tile is encoded at native resolution. In the final layer, let $k_i = x_i W_k$ be the key projection of token $i$, where $x_i$ is the token's input feature and $W_k$ the pre-trained key matrix. Every tile token attends over the union of its own tile's tokens and all $s_2$ global tokens, with weights
$$a_{ij} \propto \exp\!\Big(\frac{k_i^\top k_j}{\tau} - \frac{\lVert p_i - p_j\rVert^2}{2\sigma^2}\Big),$$
where $p_i$ is the token's centre in full-image pixel coordinates, $\tau$ is a temperature, and $\sigma$ sets the spatial falloff. This is NACLIP's key-key Gaussian attention extended across scales. Because positions are shared absolute coordinates, nearby global tokens contribute local context. Distant tokens contribute only through strong feature similarity, which is what lets a tile of white metal borrow the label "airplane" from the rest of the fuselage.
Fusion. Zoomed tiles overwrite the upsampled coarse prediction where they exist, with linear blending in overlap margins.
Peak memory is one native tile pass plus $N_g$ global tokens, roughly 784 at 448 resolution with patch size 16. It is independent of image size, and compute is linear in $B$.
Baselines
The naive reference is downsample-and-upsample. NACLIP runs on the image resized to its usual working resolution, and the mask is bilinearly upsampled. This sets the floor that every zooming method must clear on small objects and boundaries.
The main efficiency competitor is exhaustive sliding-window NACLIP at native resolution. It is the strongest single-backbone, training-free option, and its peak memory is already constant per tile. That makes it the fair test of whether our gating saves compute without losing accuracy. It runs with the same tile size, stride and CLIP weights as our zoom stage. On PC-59 at standard resolution NACLIP reports 38.4% mIoU versus SCLIP's 36.1% (NACLIP). We will reproduce those numbers before any high-resolution comparison, to confirm our implementation. Beating it means matching its mIoU at $B \le 0.4$, or exceeding it at $B = 1$ because of cross-scale context.
The strongest existing accuracy competitor is Trident, run with its released configuration on identical images and vocabularies. The file reports its relative gains (+3.5% average over ProxyCLIP, +12.0% over patch baselines on high-resolution evaluation) but no absolute per-dataset figures. There is no reported result found on Lacuna for Trident's absolute mIoU on Cityscapes or on any aerial benchmark, so we will measure it ourselves, together with its peak memory and latency on the same GPU. Beating it means statistically equal mIoU at a fraction of its peak memory. Coming within two points while running on a 12 GB consumer card would also count as a meaningful result.
MROVSeg is reported as an upper reference. It is trained, so we do not expect to beat it.
Three ablations isolate the key ingredients:
- Gate signal: the zoom stage alone, with tiles chosen by sensitivity, entropy, or random selection.
- Context: the zoom stage with cross-scale attention removed, i.e. tiles attend only to themselves.
- Coarse attention: the coarse-pass attention swapped among vanilla CLIP, SCLIP's correlative attention, and NACLIP's.
Why not just…?
Why not just downsample and run NACLIP? Small structures fall below the patch size. On SelvaBox's 4.5 cm-resolution canopy mosaics or distant Cityscapes pedestrians, a single 448 pass leaves many instances smaller than one 16-pixel patch. Chandran's 4K landmark results (Chandran) and PatchRefiner V2's 25.1% boundary-F1 gain from local refinement (PRV2) both show that coarse-only predictions lose fine geometry. Our size-binned recall curves will measure exactly how much is lost.
Why not just slide NACLIP over every tile? This is the real threat. It is simple, memory-bounded and training-free. If exhaustive tiling plus a trivial global-prior averaging matches our method at equal compute, then gating is not worth its complexity. Two experiments settle this: the accuracy-versus-compute curve over $B$, and the comparison of tiles with and without cross-scale attention at $B = 1$. Trident's 12.0% gain over segment-then-splice suggests isolated tiles leave substantial accuracy unclaimed. Our claim is that one in-network attention change recovers that context without SAM.
Why not just use Trident, or SAM masks classified by CLIP? These are likely to remain more accurate at boundaries, because SAM was trained on dense masks. The question we address is whether that accuracy requires three backbones. Our memory and latency measurements on one GPU will answer it directly. If the accuracy gap to Trident stays large at every budget, we will report that single-backbone context injection is insufficient.
Why not gate on coarse confidence, as SparseMat does? Confidence can be high and wrong for objects the coarse pass never sees, which is the tethering failure noted for SparseMat. The gate-signal ablation tests whether scale disagreement catches more of these objects. If it does not beat entropy, we will drop it.
Proposed Experiments
Datasets. Cityscapes at native 2048×1024, scored by mIoU, boundary F1, and per-instance recall binned by object area, using its instance labels. DeepGlobe (2448×2448) and Inria Aerial, evaluated zero-shot with class names as text prompts; BPT's supervised 76.6% and 77.1% mIoU (BPT) are an upper reference only. SelvaBox, with the prompt "tree crown", scoring box recall from connected components of the mask. Every run reports peak GPU memory, GFLOPs and latency on the same GPU, and we sweep $B \in \{0.1, 0.2, 0.4, 1.0\}$.
Decisive checks. There are three, each tied to a part of the claim:
- Gating. At $B = 0.4$, our method must match exhaustive sliding-window mIoU within one point on Cityscapes and DeepGlobe. Random selection at the same budget must not. If random selection matches, gating has no value.
- Gate misses. We will measure the miss rate: the fraction of ground-truth instances under 32×32 pixels that fall entirely outside zoomed tiles. The prediction is that it is clearly lower with a NACLIP or SCLIP coarse pass than with vanilla CLIP. A null result would mean that spatial-attention fixes and high-resolution routing do not interact.
- Context and scale. Cross-scale attention must beat isolated tiles at $B = 1$, especially on classes that span tiles. We also repeat everything with ViT-L. NACLIP reports that SCLIP drops 12% when moving from ViT-B to ViT-L, against 4% for NACLIP, and we test whether tiling amplifies that drop or whether shared global tokens dampen it.
References
- Ultrahigh Resolution Image/Video Matting with Spatio-Temporal Sparsity
- Attention-Driven Cropping for Very High Resolution Facial Landmark Detection
- SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference
- Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation
- Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation
- MROVSeg: Breaking the Resolution Curse of Vision-Language Models in Open-Vocabulary Semantic Segmentation
- Ultra-High Resolution Segmentation via Boundary-Enhanced Patch-Merging Transformer
- PatchRefiner V2: Fast and Lightweight Real-Domain High-Resolution Metric Depth Estimation
- SelvaBox: A high‑resolution dataset for tropical tree crown detection