Zoom, Rotate, Agree: Test-Time Augmentation That Keeps Thin Roads Alive in Zero-Shot Segmentation

General-purpose segmentation models such as the Segment Anything Model (SAM) can outline buildings, fields and water in satellite images without any training on them. On roads, they fail almost completely. A careful zero-shot study, meaning one that applies the model to a new task with no fine-tuning, found that SAM scores an intersection-over-union below 10 on DeepGlobe Roads in every prompt setting. The same study found that simply upsampling the satellite image improved SAM's accuracy, even though upsampling adds no new information (Segment Anything, From Space?). Two readings of this follow. Part of SAM's failure may be a mismatch of scale and shape rather than missing knowledge: a road network spans the whole tile and has no closed outline, while a short road segment inside a small zoomed crop looks much like the compact objects SAM was trained on. If so, the fix belongs at inference time, by showing the model many zoomed and rotated views and combining what it says.

Combining views is where thin, rare structures get hurt. Test-time augmentation (TTA) runs a model on several transformed copies of the input and averages the predictions. It is known to turn many correct predictions into wrong ones even when overall accuracy rises (Better Aggregation in Test-Time Augmentation). For a road only a few pixels wide that covers about one percent of a tile, the damage falls mostly on the rare class. Undoing a rotation or zoom requires resampling the prediction, which shifts it by a pixel or so. When shifted copies of a two-pixel line are averaged, the result is a faint, wider smear that falls below the decision threshold. The background class is never hurt this way. We propose a label-free TTA method for zero-shot thin-structure segmentation. It uses scale and orientation views chosen to fit SAM's object prior, and it combines them with a tolerance-aware, agreement-weighted vote designed so that rare classes are not erased.

Core Claim

The claim has two parts. The first is diagnostic. In dense prediction under heavy class imbalance, uniform TTA averaging corrupts the minority class far more than the majority class, and the main cause is sub-pixel misalignment after the inverse warp, not model disagreement. We will show this by extending the corruption count of When and Why Test-Time Augmentation Works from image labels to pixels, separately for each class. Here a corruption is a pixel the single-view model got right that TTA makes wrong.

The second part is constructive. Two changes recover much of SAM's lost road accuracy under a fixed budget of user clicks, with no labels and no weight updates. First, the views are zoomed, rotated crops, so each one shows SAM a locally compact road piece. Second, the vote is spatially tolerant: each view's prediction can support a pixel if it marks any pixel within a small radius of it. Each view's weight comes from how well it agrees with the other views, not from labeled data.

Written out, for pixel $p$, view $k$ of $K$, and inverse-warped road probability $\tilde y_k$, the combined road mask is

$$\hat y(p) = \mathbb{1}\Big[\textstyle\sum_{k=1}^{K} w_k \max_{q \in N_r(p)} \tilde y_k(q) \ge \tau\Big] \cdot \mathbb{1}\big[\max_k \tilde y_k(p) \ge \tau\big],$$

where $N_r(p)$ is the set of pixels within radius $r$ of $p$ and $\tau$ is a fixed threshold. The first factor lets a view vote for a road near $p$ even if its line is shifted by a pixel. The second factor keeps the output from growing wider than any single view's line. The weight $w_k$ is view $k$'s relaxed F1 against the leave-one-out consensus. Relaxed F1 counts a predicted road pixel as correct if a reference road pixel lies within $r$, which is the buffered matching already standard in road evaluation. Computing $w_k$ needs no labels.

Research Program

The work has three parts.

The first part is the measurement study. We run SAM on DeepGlobe Roads, Massachusetts and the Parcel Delineation set used by Ren et al., the last as a second thin-structure case where field boundaries are the minority class. Each dataset is run with standard flips and 90-degree rotations combined by mean averaging. We record per-class corruption and correction rates. We then repeat the run with views that are only shifted by whole pixels, which needs no resampling. If corruption of the road class drops sharply under pure shifts, misalignment is confirmed as the cause. If it does not, the corruption reflects genuine disagreement between views and the tolerant vote is the wrong remedy. In that case we would report it and redirect the method toward agreement weighting alone.

The second part builds the view set and the prompting. Following the resolution bias in Segment Anything, From Space?, each view is a crop at one of several zoom factors and one of several rotations, upsampled to SAM's native input size. The rotations are chosen so that the local road direction in each crop takes several different angles. Where we have a measured direction, it comes from the structure tensor, a simple gradient statistic that gives local edge orientation.

Prompting follows Ren et al.'s interactive-annotation protocol, which uses a fixed number of user clicks per tile. Clicks that fall inside a crop become that crop's prompts. Crops without a click receive at most two points sampled along the skeleton of the current consensus mask, using SAMAug's maximum-distance rule (SAMAug: Point Prompt Augmentation for Segment Anything Model). The cap of two follows directly from SAMAug's finding that adding more than two points to one mask degrades it. Our design spreads prompts across many small views instead of piling them onto one mask.

One constraint pulls against zooming. In LiDAR power-corridor classification, rebalancing the data by resampling destroyed the surrounding context and lowered F1 by more than 5% (Systematic Comparison of Power Corridor Classification Methods from ALS Point Clouds). Zooming into a crop also throws away context. For that reason the view set always includes coarse, wide views alongside the zoomed ones. Ablations then test whether the coarse views earn their cost.

The third part is the aggregation rule above. We test it against uniform averaging, pixel-wise majority voting (Lightweight Encoder-Decoder Architecture for Foot Ulcer Segmentation), and an uncertainty-weighted merge in the spirit of the pixelwise merging in ZODS-RS. ZODS-RS is the closest training-free remote-sensing pipeline. It handles compact oriented objects and is limited on tiny targets, while our method targets open linear networks.

Baselines

The naive reference point is single-pass SAM on the native tile with the same clicks. Ren et al. report an IoU below 10 for this setting on DeepGlobe Roads in Segment Anything, From Space?.

The strongest zero-shot competitor is the best single view. This means SAM run on upsampled, tiled crops with the same clicks, using the single zoom factor that does best on a held-out set of tiles. We expect this to be a real threat, because Ren et al.'s upsampling finding suggests much of the gain may come from scale alone. No reported result found on Lacuna for upsampled or tiled SAM on DeepGlobe Roads. We will therefore run it ourselves on identical tiles. It gets the same click budget, and it gets a compute budget matched in total forward passes by using finer tiling, so that any gain from our method cannot come from extra compute. Beating it means a road IoU gain larger than the spread across five random click-placement seeds, together with fewer skeleton breaks per kilometre of road.

The second competitor is SAMAug applied directly to full tiles. The paper reports Dice gains of 0.01 to 0.07 on natural and medical images, but no reported result found on Lacuna for SAMAug on overhead roads.

The third competitor is uniform-mean TTA with an oracle-tuned threshold. Its threshold is chosen on labeled tiles, which is a deliberate unfair advantage. It isolates whether our vote does more than recalibrate the threshold.

Two ablations isolate the key ingredients. One keeps our views but swaps the tolerant vote for uniform averaging. The other keeps the tolerant vote but uses only flips and 90-degree rotations, with no zoom.

As an upper reference for the aggregation step, we fit dense ClassTTA-style weights, one weight per view and class, on a small labeled set, following Better Aggregation in Test-Time Augmentation. This is not zero-shot. It shows how much of the gap from labeled weighting our label-free weights close.

Fully supervised road networks give the ceiling: PWFNet reports 70.52% IoU on DeepGlobe (PWFNet), DPSDA-Net reports 64.46% IoU on Massachusetts (DPSDA-Net), and the DeepGlobe focal-loss baseline reports 0.545 IoU (DeepGlobe 2018). We do not expect to match these numbers. Our stated target is to close at least half the gap between zero-shot SAM and the 0.545 baseline.

Why not just…?

Why not just upsample the image and run SAM once? This is the most serious alternative. Its premise is Ren et al.'s own evidence that SAM has a resolution bias. A single scale cannot be right everywhere, though: rural tracks and urban arterials differ in width, and a large zoom factor removes the context that the power-corridor study found necessary. The matched-compute best-single-view baseline settles the question directly. If it matches our method on IoU and on breaks per kilometre, we will report that multi-view aggregation adds nothing beyond scale, and the contribution reduces to the corruption diagnostic.

Why not just use standard flip-and-rotate TTA and lower the threshold? If minority shrinkage were only a calibration problem, a lower threshold would fix it. The oracle-threshold baseline is that fix, given labels it is not entitled to. The pure-shift experiment separates misalignment from disagreement. A tuned threshold cannot fix line widening or broken connectivity, which is why we report topology alongside IoU.

Why not just fine-tune SAM on roads, as SAM-Road does? Segment Anything Model for Road Network Graph Extraction fine-tunes SAM's image encoder and adds a graph network, reaching 90.47% TOPO precision on City-scale, a measure of how correctly the extracted road graph is connected. That answers a different question. It requires labeled roads from the target region, and DeepGlobe shows models trained on one city's data dropping from 0.885 to 0.544 F1 in another. Our method targets regions without labels. The supervised numbers serve as ceilings, not as competitors.

Proposed Experiments

The main experiment runs the full method against every baseline under the same click protocol on DeepGlobe Roads, Massachusetts and LRSNY. We report IoU, relaxed F1, per-class corruption and correction rates, and skeleton breaks per kilometre of road. Cross-region robustness is tested on LSRV and CHN6-CUG with zoom and rotation settings frozen from DeepGlobe, and Parcel Delineation serves as a non-road thin class.

Three results would be decisive. First, the pure-shift experiment must show that misalignment drives road-class corruption. Second, the tolerant vote must cut road corruption relative to uniform averaging on identical views without raising background corruption. Third, the full method must beat the matched-compute best-single-view baseline beyond click-seed variance. If the second result fails, the aggregation idea is wrong. If only the third fails, scale alone is the story, and we will say so.