Scale-Warped Rotary Coordinates for Lab-to-Field Vision

Vision models trained on tidy "laboratory" photos, such as a single leaf centered on a gray table, fail badly when deployed in the field. There, the same disease appears on a leaf a meter away, a lesion covers 1% of the frame, and near and far plants share one image. This note proposes a fix inside the transformer's position encoding.

Rotary position embedding (RoPE) is a way of telling attention where image patches sit by rotating each patch's query and key vectors by an angle proportional to its coordinate. The attention score between two patches then depends only on the difference of their coordinates, and nothing forces those coordinates to be the integer patch grid. Our idea is to give every patch a continuous, content-dependent coordinate. The coordinate is obtained by summing up the inverse of a predicted local object scale, so that "two patches apart on a distant leaf" looks to attention like "eight patches apart on a lab leaf." The network then meets field images at the spatial scale it learned in the lab. This costs one small convolutional head and no extra attention compute.

The opportunity is that rotary attention is already relative and invertible: its geometry can be edited per token without retraining the backbone's attention weights. Field shifts in agriculture and driving are dominated by scale and framing changes, not only by texture corruptions. Lab training fixes the scale of objects in patch units. The open question is whether correcting that scale mismatch at the level of positions recovers accuracy that more data augmentation and multi-scale testing cannot recover at equal cost.

Core Claim

A 2D rotary vision transformer trained on lab-scale data learns attention patterns tuned to object extents measured in patches. Under field scale shift these patterns become mismatched, which is a position-level failure distinct from a feature-level failure. Replacing the integer patch grid with coordinates warped by a predicted inverse-scale field restores the lab-time relationship between relative distance and object extent. Concretely, with $\hat s_k$ the predicted local scale at patch $k$ (bounded to $[1/4, 4]$) and patches indexed along a row, the horizontal coordinate of patch $i$ is

$$p_i^x = \sum_{k \le i} \frac{1}{\hat s_k},$$

and analogously along columns for $p_i^y$. These coordinates feed unchanged into a RoPE-Mixed attention layer. Since attention depends only on $p_i - p_j$, distances shrink or stretch locally. Patch ordering is preserved because every increment is positive.

We predict three things:

  • Small targets. Gains concentrate on small and distant targets, where the lab-to-field scale ratio is largest.
  • Beyond resizing. Warping beats global image resizing at matched compute, because one image contains many scales.
  • Beyond augmentation. Warping beats training with scale augmentation alone.

How it relates to prior work

Two papers supply the mechanism.

  • Rotary Position Embedding for Vision Transformer introduced RoPE-Mixed, a 2D rotary encoding with learnable frequencies that avoids the axis-aligned artifacts of separate x and y rotations. It reported +2.5 mIoU on ADE20k and +1.8 box AP on MS-COCO for ViT-B. That work evaluates only clean benchmarks, studies resolution extrapolation, and explicitly leaves interpolation (inputs effectively smaller than training) unresolved. Field zoom-out is exactly that regime. We keep its encoding and change only the coordinates it receives.
  • Selective Rotary Position Embedding makes 1D rotation angles input-dependent via a short convolution followed by a cumulative sum, for state tracking in language-style sequence models. We borrow the convolution-then-cumulative-sum construction and move it to 2D images, where the quantity summed is inverse object scale. That paper also reports gradient spikes when learnable phase terms are added, which constrains our design: we bound $\hat s$ and add no phase gate.

LieRE maps continuous coordinates to rotations through a matrix exponential. It beat RoPE-Mixed on CIFAR-100 (70.3% vs 68.8%). We treat it as a second backbone on which warped coordinates can be dropped in, not as a competitor. Its evaluation, like RoPE-Mixed's, never leaves curated data.

Three works motivate the problem.

  • WildDash showed a Cityscapes-trained segmenter falling from about 64 to 16.5 mIoU on hazard-rich dashcam frames, with only 30.7 mIoU after training on the more diverse Mapillary Vistas. Diversity alone leaves most of the gap.
  • PlantSeg found that in-the-wild segmenters fail on lesions occupying about 1% of the image and that performance tracks lesion compactness. This is the scale-dominated failure we target.
  • Adaptive Attention Distillation built the ER-FSS benchmark, which pairs clean lab support images with perturbed field queries; "small scale" is one of its perturbation types. It attacked "attention drift" with class queries on standard backbones. Our approach is complementary: it changes the geometry attention operates in rather than adding query-level guidance.

One line of work could be read as contradicting us. DoPE and VideoRoPE show that particular rotary frequency bands cause attention sinks or "hash collisions" in long 1D and video sequences. That suggests the fix might be frequency surgery rather than coordinate warping. We test this directly with a frequency-band ablation (below) instead of assuming it away.

Research Program

We build the following components.

  • Backbone. A ViT-B segmentation model with RoPE-Mixed attention, using a UperNet head as in the RoPE-Mixed paper.
  • Scale head. A scale predictor consisting of two 3×3 convolutions on the patch embeddings with a bounded sigmoid output. It produces $\hat s$, which is turned into coordinates by row-wise and column-wise cumulative sums before the first attention block. The separable sums are an approximation to integrating a 2D scale field. We report how much this matters by comparing against an oracle warp computed from ground-truth object masks.
  • Training signal. During lab training, image crops are randomly zoomed by a known factor $z \in [1/4, 1]$. An auxiliary loss asks $\hat s$ to equal $z$ on the zoomed region, giving the scale head direct supervision with no field labels. The task loss trains everything else.

We then analyze the model in three ways.

  • Attention distances. We measure mean attention distance per layer, as in the RoPE-Mixed analysis. Our hypothesis is that field images under the unwarped model show attention distances that are too long relative to object extent, and that warping restores lab-like distances.
  • Scale calibration. We check whether $\hat s$ correlates with true object size on PlantSeg masks.
  • Orientation diagnostic. As a cheap side diagnostic, we test whether the upside-down failure documented by WildDash persists in rotary models. Rotary attention carries no absolute position except at image borders. If the failure persists, it lives in content features and is outside this proposal's mechanism; if it vanishes, the WildDash failure was largely an absolute-encoding artifact.

Baselines

Naive reference. The naive reference is the same ViT-B with standard RoPE-Mixed on the integer grid, trained with ordinary flips, crops and color jitter. It establishes the size of the lab-to-field drop for rotary models, which no paper in our file reports.

Strongest augmentation competitor. The strongest practical competitor is the identical RoPE-Mixed model trained with exactly our random-zoom augmentation but with coordinates left unwarped. This is the most direct threat, because the network may simply learn scale-robust features from the augmentation. It receives the same data, epochs, and hyperparameter search budget, which is 12 configurations for every arm. Beating it means at least 2 mIoU on the small-lesion stratum of PlantSeg and on WildDash with no loss on lab validation. Below that we would not claim the warp matters.

Strongest compute-unconstrained competitor. This is test-time multi-scale inference: a resize pyramid at 1×, 1.5× and 2× with prediction fusion, applied to the augmented model. We report it both at its native cost and against a warped model given matched FLOPs.

Global-warp ablation. This ablation replaces the per-patch $\hat s$ with a single image-level scalar, analogous to positional interpolation in language models. It isolates whether per-token warping, rather than global rescaling, is the key ingredient.

Oracle warp. An oracle warp built from ground-truth object sizes gives the ceiling.

Absolute and published references. We include an absolute-position ViT and a ConvNeXt-L for continuity with published numbers. On PlantSeg, PlantSeg reports ConvNeXt-L at 46.24% mIoU, and our in-field training arm must match or exceed this to be credible. On WildDash, the reported references are 16.5 mIoU (Cityscapes-trained DRN-D38) and 30.7 mIoU (Mapillary-trained), both from WildDash. These reflect an older architecture, so we retrain our own Cityscapes-only baselines rather than compare to them directly. On ER-FSS, Adaptive Attention Distillation reports a 3.3–8.75% average mIoU improvement over prior methods, but no absolute per-dataset number is found in the file. We will run its released setup if available and otherwise report "no reported result found on Lacuna" alongside our own reruns.

Why not just…?

Why not just train with heavy scale augmentation? This is the real threat, and the second baseline above is built to settle it. The argument for warping is structural. Augmentation must teach every attention head to be correct over a 16× range of object extents with fixed weights, while warping lets heads keep one learned extent and moves the inputs. Existing evidence is only suggestive: Mapillary-scale diversity still left WildDash at 30.7 mIoU. If the augmented unwarped model matches ours within 2 mIoU on small-lesion strata, we will report that positional warping adds nothing and stop.

Why not just resize the image at test time? Resizing applies one scale to the whole image, but field frames mix near and far objects. Upsampling also multiplies attention cost quadratically, which matters for the edge platforms where TomFormer already reports latency problems. The matched-FLOP comparison shows whether this argument holds. A multi-scale pyramid winning at matched cost would refute our efficiency claim, though not necessarily the accuracy claim.

Why not just fix the rotary frequencies, as DoPE and VideoRoPE do? Those works found specific frequency bands harmful in long sequences. A field failure might likewise come from high-frequency spatial channels reacting to clutter rather than from scale mismatch. We test this by masking the lowest or highest quarter of RoPE-Mixed frequencies at test time, training-free in the style of DoPE-by-parts. If band masking recovers as much as warping on WildDash, the problem is spectral, not geometric, and the proposal's premise is wrong.

Why not just use a CNN or add attention modules like CBAM? PlantSeg reports MobileNetV3 at only 10.22% mIoU against 46.24% for ConvNeXt-L in the wild. Its authors attribute part of the gap to self-attention's handling of scattered lesions. CBAM-style gating also does not address scale. We therefore include ConvNeXt-L as a reference but not as the main competitor.

Proposed Experiments

Experiment 1: controlled zoom-out on lab data. We train on PlantVillage-style lab segmentation crops and test on the same images synthetically zoomed out by 2–4× and pasted onto PlantSeg backgrounds. Scale is known exactly, so this measures how much of the drop the warp recovers relative to the oracle.

Experiment 2: real field transfer in two settings.

  • PlantSeg. We train on PlantSeg and evaluate stratified by lesion area and convex-hull ratio, predicting gains concentrated below 5% of image area.
  • Cityscapes to WildDash. We train on Cityscapes and evaluate on WildDash, reporting mIoU per hazard cluster and on negative test cases. We expect no gain on glare or overexposure, which are not scale hazards. This is a built-in specificity check: uniform gains across all hazards would point to a regularization effect rather than the claimed mechanism.

Experiment 3: ER-FSS. We evaluate on ER-FSS with the warped backbone in place of AAD's backbone, reporting the "small scale" query subset separately.

Decisive checks. These run across all three experiments. First, the per-token warp must beat both the scale-augmented unwarped model and the global-warp ablation on small-object strata. Second, predicted $\hat s$ must correlate with true object size on PlantSeg masks; if it does not, any gain cannot be attributed to scale correction. Third, attention-distance profiles on field images under warping should move toward lab profiles. Fourth, frequency-band masking must not reproduce the gain.

Outcomes. If all four checks hold, scale-warped coordinates are a cheap, backbone-agnostic route to lab-to-field transfer. If the augmentation baseline ties, we will have established the first measurement of how rotary vision models degrade in the field and that position geometry is not where the gap lives.