Interactive segmentation tools let a person cut an object out of an image by clicking on it. Current tools hit two problems on large images. They shrink the photo to about one megapixel before processing, which erases wires, spokes and hairline cracks. And the part of the network that reads the image, a vision transformer, gets far more expensive as resolution grows: it compares every image patch with every other patch, so doubling the side length multiplies that comparison work by sixteen. Most of that work is wasted. Once a coarse mask exists, the only pixels whose labels are still in doubt lie in a thin band around the mask's edge. That band grows with the object's perimeter, not with the image's area.
We propose to encode a high-resolution image in two layers. A cheap global pass runs once at low resolution, and its outputs are kept in memory. A detailed pass then computes full-resolution patch features only inside the current band of uncertainty, and each detailed patch also looks at the stored low-resolution features for global context. Each new click moves the band, so only newly uncovered patches are computed. Patches computed earlier are reused, because they describe the image and not the prompt. The goal is the click efficiency of today's best high-resolution interactive models at a fraction of their time-to-first-mask and memory. A further goal is a cost per click that stays bounded as images grow beyond 4K.
Core Claim
The claim has two parts. First, in interactive segmentation the user's clicks and the model's current mask together show where expensive high-resolution computation is needed. Second, this can be exploited inside the image encoder without giving up the main efficiency trick of Segment Anything-style tools: compute the image features once, then answer every click with a light decoder. The two parts conflict for any prompt-conditioned encoder. If the features depend on the clicks, the encoder must be rerun on every click. We resolve this by keeping the features prompt-agnostic and making only the choice of which patches to compute depend on the prompt. The encoder becomes a lazily filled cache of full-resolution features. The clicks decide where to fill it, and its contents never need recomputing.
If $M$ is the number of full-resolution patches inside the band, $w$ the size of each patch's local attention neighbourhood, and $N_c$ the number of stored low-resolution patches, then each newly activated patch costs on the order of $w + N_c$ attention operations. The total encoder cost of a session is
$$C_{\text{session}} \approx N_c^2 + M_{\text{total}}\,(w + N_c),$$
where $M_{\text{total}}$ counts every distinct patch activated across all clicks. A uniform high-resolution encoder pays for all $N$ full-resolution patches instead. The band's area is roughly its width times the boundary length, so for compact objects $M_{\text{total}} \ll N$. For fences, foliage and nets the boundary can fill much of the frame, and the method's advantage should then shrink toward zero. Measuring where that crossover happens is part of the contribution, not a footnote.
Relation to Prior Work
The motivation comes from three findings. Segment Anything fixes its encoder at $1024 \times 1024$. NTClick and Deep Interactive Thin Object Selection show that this downsampling puts a ceiling on thin-structure accuracy that more clicks cannot lift. VLTP and HRSAM report that the image encoder accounts for over 90% of interactive latency.
The closest systems share our goal but spend computation uniformly. HRSAM swaps global attention for block-diagonal local windows plus a linear-time state-space scan, so it can be trained at $1024$ and run at $2048$. It reports degradation at $4096$, and every patch in the grid is still processed. NTClick reaches $4096$ by cascading two networks. A coarse $448 \times 448$ network outputs a foreground/background/uncertain map, and a second network then runs neighbourhood and grid attention over the whole high-resolution image. We reuse NTClick's three-way uncertainty map as our selection signal, but we spend high-resolution compute only where that map says "uncertain," in one network that shares a feature cache. RWKV-SAM replaces attention with a linear recurrent backbone and reaches 80.5 mIoU on DIS, but its authors list ultra-thin structures as an open failure. That is a constraint on our design: we keep softmax attention for the detailed patches rather than linearising it.
FocSAM is the closest in spirit. Its windowed attention runs only on windows that intersect the predicted mask, and this fixes the problem of masks collapsing after a corrective click. However, it works on the cached, fixed-resolution SAM features, after encoding, so it cannot recover detail the encoder never saw. We move the mask-directed selection into the encoder and to native resolution.
The token-pruning literature supplies the mechanics. DToP stops computing "easy" patches early in semantic segmentation and shows that keeping a few context patches matters. We keep the entire coarse layer as context for the same reason. VLTP prunes encoder patches using a language token and lets pruned patches be reactivated. Our band movement is a spatial, click-driven form of reactivation. Adaptive patch sizing (APT) varies patch size by image entropy and needs no prompt. We argue entropy is the wrong signal here, because textured background is high-entropy but irrelevant, and we test that directly. We borrow APT's evidence that mixed-scale patches can be fine-tuned from a pretrained checkpoint in about one epoch.
Research Program
We will build the system on a ViT-Base backbone in three parts.
The coarse layer is a standard windowed ViT run at about $1024$ on the downsampled image. Its patch features are stored in memory, and a light SAM-style decoder turns them plus the clicks into a mask and a three-way uncertainty map.
The detailed layer uses the same transformer blocks at native resolution, run only on the selected patches. Each selected patch attends to its active full-resolution neighbours within a local window and to all stored coarse patches; the coarse patches are not updated. Because the input to the detailed layer is image-only, its outputs can be stored and reused across clicks. In one ablation, a stored full-resolution patch is recomputed when a newly activated neighbour enters its window; we measure whether this "stale neighbourhood" effect matters.
The selection rule activates full-resolution patches inside the uncertain region and within a small radius of every click. Clicks act as forced activation, which also suits NTClick-style noisy clicks near thin parts. A fixed per-click token budget makes latency predictable, and the budget is filled by uncertainty rank.
Training uses simulated iterative clicking, as in Iteratively Trained Interactive Segmentation. It combines a mask loss with feature distillation at the selected patches from a uniform high-resolution teacher. We will use a dense windowed ViT run at full resolution, offline, where memory permits, so that the student learns to match dense features from sparse context. Training data follow NTClick (HQSeg-44K) so that comparisons are like-for-like.
The analysis program asks three questions. How large does $M_{\text{total}}$ grow relative to $N$ as clicks accumulate, across object types? Does accuracy at a selected patch depend on how many of its neighbours are active? And does the click-and-uncertainty signal beat image entropy at an equal patch budget?
Baselines
The naive reference is SAM at $1024 \times 1024$ with no high-resolution path. Mamba or RWKV reports it at 62.0 mIoU on DIS, and NTClick reports SAM-HQ at 11.04 NoC@90 on DIS5K-VD. Any proposal that does not clearly beat these has failed.
The strongest accuracy competitor is NTClick, which reports 7.23 NoC@90 on DIS5K-VD (NoC@90 is the number of clicks needed to reach 90% IoU), against 10.35 for SegNext. It also reports a NoC@90 gain of 5.46 and a 5-click mIoU gain of 8.04% on DIS5K-TE4 over previous leaders. It is the strongest competitor because it is the only cited system that already runs interactively at $4096$ with full-resolution attention. We will retrain it, or use released weights if available, on the same HQSeg-44K training data. Both systems will be measured on the same GPU and run with the same click simulator. Beating it means matching its NoC@90 within 0.3 clicks on DIS5K-VD and TE1–TE4 while cutting time-to-first-mask and peak memory by at least half. Lower NoC at equal cost would be a bonus, not the core claim.
The strongest efficiency competitor is HRSAM, which reports 10–17 ms per click on an RTX 3090 and 38% of SegNext's latency. That per-click figure excludes the one-time encoding of the whole image, and that one-time encoding is exactly where we expect to win. We will report time-to-first-mask, per-click latency and total session time side by side. Beating HRSAM means lower time-to-first-mask and session time at $2048$–$4096$ with equal or better NoC@95.
FocSAM is the decoder-side competitor. It reports a mean NoC@90 of 5.38 across its benchmarks, against SAM's 6.85, at 0.39 s per click on CPU. Its benchmarks are general (GrabCut, Berkeley, DAVIS and others), so we will compare on DAVIS and on DIS5K at matched backbone size.
Two ablations isolate the key ingredients. The first holds the patch budget fixed and swaps the selection signal among uncertainty plus clicks, clicks alone, APT-style entropy, and random selection. The second removes the coarse-context attention, so full-resolution patches see only local neighbours.
Why not just…?
Why not just crop around the mask and run a high-resolution model on the crop? This is the most serious threat, because selecting the uncertainty band is a soft version of cropping. The differences are testable. Our full-resolution patches still attend to global coarse context. The band can be non-convex and broken into pieces, such as a bicycle's spokes and frame, where a bounding crop is mostly wasted interior. And patches computed after one click are reused after the next. We will implement a crop-and-refine baseline that uses the same detailed-layer weights, gives each crop an equal patch budget, and fits one crop per connected uncertain region. If it matches us on NoC and cost, the paper's claim reduces to "crops suffice." We would then report that result and narrow the claim to the cross-click reuse alone.
Why not just run an efficient uniform encoder like HRSAM once and cache it? Per-click cost is already low for HRSAM, so our advantage must show up in time-to-first-mask, memory, and resolution beyond its reported degradation at $4096$. If HRSAM's one-time cost at $4096$ turns out acceptable and its accuracy holds, our method's value is confined to larger images and memory-limited hardware, and we will say so. The resolution sweep in the experiments settles this.
Why not just prune by image entropy or saliency, as APT and FastV do? APT's own authors note that their entropy heuristic is not aligned with the downstream task. VLTP's gains come from tying pruning to the target, which suggests intent matters more than texture. The equal-budget selection ablation measures this directly. If entropy selection matches ours, the click-driven part of the proposal is unnecessary.
Why not just distill to a smaller backbone like EfficientSAM? A smaller ViT still compares all patches with all patches, so its cost still grows quadratically with resolution. Distillation also leaves the resolution ceiling in place unless the student runs at full resolution. We will include a ViT-Tiny uniform encoder at $2048$ to show where its cost curve crosses ours.
Proposed Experiments
Accuracy will be measured on DIS5K-VD and TE1–TE4 (NoC@90, NoC@95, 5-click mIoU, and the human-correction-effort score), ThinObject-5K and HRSOD (IoU on thin regions, $IoU_{thin}$), and DAVIS (NoC@90). Efficiency will be measured on one GPU as time-to-first-mask, mean seconds per click, total session time to 90% IoU, and peak memory. We will also record $M_{\text{total}}/N$ for every session.
Three decisive checks follow. The first is a resolution sweep from $1024$ to $4096$ on DIS5K. It succeeds if our session cost grows clearly more slowly than HRSAM's and NTClick's while NoC stays within the stated margin of NTClick. A further stress test resamples DIS5K images upward; this tests scaling only, not accuracy on real 8K content. The second is a breakdown by object type. We predict $M_{\text{total}}/N$ is small for compact objects and approaches one for fences and nets, and if the advantage vanishes on most of DIS5K the claim fails. The third is the crop-and-refine comparison and the selection ablation described above, which decide whether the gain comes from global context, cross-click reuse, or merely from spending compute near the object.
References
- Segment Anything
- NTClick: Achieving Precise Interactive Segmentation With Noise-tolerant Clicks
- Deep Interactive Thin Object Selection
- VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
- HRSAM: Efficient Interactive Segmentation in High-Resolution Images
- Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Model
- FocSAM: Delving Deeply into Focused Objects in Segmenting Anything
- Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation
- Accelerating Vision Transformers with Adaptive Patch Sizes
- Iteratively Trained Interactive Segmentation