Dense image matchers built on attention find pixel correspondences between two photos by letting every location in one image look at every location in the other. Examples are LoFTR, which matches without a separate keypoint detector, and its faster successor Efficient LoFTR. These matchers work well when the two photos are roughly upright and at similar zoom, and they degrade badly once the second photo is rotated or rescaled. The usual fix is to make features rotation-aware by giving each pixel a copy of its feature for every one of $K$ orientations and $S$ scales. That multiplies the number of tokens by $KS$. Since attention cost grows with the square of the token count, the fix also multiplies the cost by $(KS)^2$. Rotation robustness and attention cost are therefore one problem, not two.
We propose that the extra orientation and scale axes do not need to be paid for quadratically. Relative rotation between two images shows up as a cyclic shift along the orientation axis, and relative scale shows up as a shift along the log-scale axis. A linear-attention summary of the second image can be cross-correlated against each query along these axes for all $KS$ candidate shifts at once with a Fast Fourier Transform (FFT). The result is a single matcher that estimates rotation and scale internally and continuously, at roughly the cost of $\log(KS)$ extra work per token. Current systems instead rectify the images upstream with separate networks, or run the matcher once per guessed rotation.
Core Claim
A dense matcher whose cross-attention is linear in the number of spatial tokens and an FFT correlation along the orientation and scale axes can do three things together. It can match image pairs under arbitrary in-plane rotation and large scale change. It can recover angles between the discrete orientation bins, which removes the accuracy dips that discrete equivariant networks show. And it runs in a single forward pass whose cost grows as $O(N \cdot KS\log(KS))$ rather than the $O((NKS)^2)$ of attention over the full rotation- and scale-expanded token set. Here $N$ is the number of spatial tokens, $K$ the number of sampled orientations, and $S$ the number of sampled scales.
The new element is putting the relative-pose search inside attention as a correlation that can be computed in the frequency domain. This differs from equivariant convolution with pooling, from upstream rectification, and from test-time search over rotations.
Where This Sits
Three papers motivate the gap.
LoFTR interleaves self- and cross-attention on a 1/8-resolution grid and uses kernelized linear attention to keep cost linear in $N$. It reports 65.9% AUC at 3px on HPatches. Efficient LoFTR pools tokens up to 16× before attention, runs about 2.5× faster, and reaches 56.4 AUC@5° on MegaDepth. Both rely on Cartesian grids and absolute positional encodings, and both assume roughly aligned inputs. SRTPN documents that LoFTR fails under large rotation and scale on multimodal remote sensing imagery. Its workaround is a ResNet18 plus FFT scale regressor and a ResNet101 classifier that assigns the rotation to one of 8 bins spaced 30° apart (the file reports both figures, although 8 × 30° covers only 240°). The rectified images then go to LoFTR. SRTPN is the closest existing work. It leaves three things open, all acknowledged in the paper: residual rotation error from the discrete bins, three networks run in sequence, and a heavy matching stage.
Two papers supply the method.
ReF uses steerable convolutions over 8 orientations. These are convolutions whose output channels shift cyclically when the input rotates. ReF reaches a peak 0.469 mean matching accuracy (MMA) at 3px on HPatches, against 0.212 for the augmentation-trained RoRD. We reuse its steerable backbone.
Polar Transformer Networks show that in log-polar coordinates rotation and scaling become translations. We reuse this idea along the feature axes rather than the image axes. That lets us avoid its main restriction, a single global origin for the whole image.
Three papers constrain the design.
ReF reports that adding ordinary convolutions after its orientation pooling destroys invariance. It also shows accuracy dips between multiples of 45°. We therefore keep the orientation axis alive through attention instead of pooling it away, and we read off sub-bin angles from the Fourier phase. Fourier Image Transformer orders 2D Fourier coefficients into a sequence and suffers high-frequency artifacts. We therefore apply the FFT only along short, band-limited group axes ($K$ about 16), never along the long spatial sequence.
Two papers supply sub-quadratic precedents, and our use differs from both. FMLP-Rec replaces attention with learnable FFT filters, which gives 1D shift equivariance along the sequence index. Monarch Mixer uses FFT-like structured matrices with no geometric meaning. We use the FFT for a different purpose: the axis it runs over is a transformation group, not token position.
Research Program
The backbone is a steerable convolutional network. It produces, at each coarse location $i$, features $q_i[g,s]$ indexed by orientation $g \in \mathbb{Z}_K$ and log-scale level $s \in \{1,\dots,S\}$, built from an image pyramid. Absolute positional encodings are removed, because they break rotation equivariance. Self-attention instead uses a relative encoding based only on distance between tokens.
Cross-attention from image A to image B uses a kernel feature map $\phi$, as in LoFTR, so each image needs one summary per group element:
$$M[h] = \sum_j \phi(k_j[h])\, v_j^{\top}, \qquad o_i(r) = \sum_{g} \phi(q_i[g])^{\top} M[g+r].$$
Here $h$ and $g$ index orientation–scale cells, $j$ runs over tokens of image B, $v_j$ is the value vector of token $j$ in image B, and $r$ is a candidate relative shift. The index $g+r$ wraps cyclically in orientation and is zero-padded in scale. Because $o_i(r)$ is a cross-correlation over $g$, all $KS$ shifts are obtained with one FFT per channel.
The model then forms a distribution over shifts from the matching energy (normalized analogously to $o_i(r)$, omitted for brevity), and uses it in two ways. Each token gets its own soft shift weights. A global consensus pooled over all tokens gives the pair-level rotation and scale. A learned gate mixes the two, which handles both pure image-level rotation (satellite and drone imagery) and locally varying pose (wide-baseline scenes).
Continuous angles come from treating the orientation axis as band-limited. The phase of the first angular harmonic of the consensus gives the rotation at sub-bin resolution. The estimated rotation and scale then warp the fine-level crops, and the fine-level refinement follows Efficient LoFTR unchanged.
Training uses MegaDepth pairs with synthetic rotations over the full 0–360° range and scale factors of 0.5–2×. The losses are LoFTR's matching losses plus a loss on the recovered relative pose. Where ground truth rotation is available, it supervises the shift estimate.
Baselines
The naive reference point is Efficient LoFTR as released, trained without rotation augmentation. Its reported MegaDepth relative-pose AUC@5° is 56.4 Efficient LoFTR. LoFTR reports 52.8 LoFTR. The reason to include it is to measure how far a strong but unprotected matcher collapses as rotation grows. Beating it means flat accuracy across angles, while staying within about one AUC point of 56.4 on unrotated MegaDepth.
The strongest learned competitor on aligned data is Efficient LoFTR retrained with the same rotation and scale augmentation, data and schedule we use. Every learned baseline gets the same tuning budget and runs on the same GPU.
The strongest existing method for the rotated-and-scaled regime is the SRTPN pipeline SRTPN. It reports 115 of 142 effective matching pairs on Google Earth, 1.038 px RMSE on VIS-NIR, and 1.236 px average RMSE across its sets. We evaluate on its four datasets with its protocol. Beating it means more effective pairs and lower RMSE, from one network, at lower total latency.
For sparse rotation-invariant matching we include ReF, which reports 0.469 peak MMA at 3px on HPatches. We compare the full curve of MMA against rotation angle, not only the peak. We include 3MRS as a classical reference on CoFSM 3MRS, with reported RMSE 1.47 px and a 100% success ratio.
Three ablations isolate the key ingredient.
The first replaces our orientation axis with ReF-style max pooling before attention. This tests whether keeping the group axis through attention matters.
The second computes $o_i(r)$ by looping over shifts instead of using the FFT. The numbers are identical, so this measures only cost.
The third runs full softmax attention over all $NKS$ expanded tokens on small images. This gives a quality ceiling and shows whether linearizing the attention costs accuracy.
Why not just…?
Why not just train Efficient LoFTR with heavy rotation and scale augmentation? This is the cheapest option and must be run. The evidence against relying on it is indirect. In ReF's comparison, the augmentation-trained RoRD reached 0.212 peak MMA against 0.469 for architectural equivariance. 3MRS reports that learned and handcrafted descriptors break down beyond about 20° of rotation or 30% scale change. Neither result concerns attention matchers specifically, so the augmented-Efficient-LoFTR baseline settles the question directly. If it matches our curve on rotated HPatches and rotated MegaDepth, the proposal's accuracy claim fails, and only an efficiency claim remains.
Why not just run the matcher on K rotated copies of the second image and keep the best? This test-time search is the most serious threat. It is simple, and it is exactly what our FFT step computes, but done by brute force. We expect equal or better accuracy at the bin angles, at about $K$ times the cost and with no sub-bin resolution unless $K$ grows. We run it with $K$ equal to 8, 16 and 36 and plot accuracy against wall-clock time. Our method must lie on a better tradeoff curve, especially at angles between bins. If brute-force search with 36 rotations is both cheaper in practice and as accurate, we would report that honestly. Scale search multiplies the cost again by $S$, so this baseline should lose most clearly on the combined rotation-and-scale test.
Why not just rectify first, as SRTPN does, or with a spatial transformer? Upstream rectification is ill-posed when pose varies across the scene. Polar Transformer Networks report that a plain spatial transformer reaches 12.35% error on SIM2MNIST, no better than a CNN, and that log-polar warping needs a single global origin. SRTPN's 30° bins leave up to 15° of residual error that LoFTR cannot correct. Our gate between per-token and global shift estimates is designed to cover the local case. The SRTPN comparison, and the ablation that forces global-only shift estimation, test whether this matters.
Proposed Experiments
The decisive experiment is the rotation curve. On HPatches and on MegaDepth we sweep in-plane rotation from 0° to 360° in 1° steps and scale from 0.5× to 2×. We report MMA at 3px and relative-pose AUC@5° at each setting, against every baseline above. Success means a flat curve with no dips at the midpoints between bins, which is where ReF dips. Further, the curve should sit above both augmented Efficient LoFTR and the 8-rotation test-time search away from the bin angles.
The second experiment covers the remote sensing application: the SRTPN datasets and CoFSM, scored by effective pairs, correct-match count, and RMSE.
The third covers efficiency. We measure runtime and peak memory as functions of $N$, $K$ and $S$, comparing against full attention over expanded tokens and against test-time search. The prediction is near-linear growth in $N$ and roughly $K\log K$ growth in $K$, which follows directly from the FFT correlation in the equation above.
Finally, on unrotated MegaDepth and ScanNet, we check that robustness costs little. The ScanNet reference is LoFTR's reported 22.06 AUC@5°. If unrotated MegaDepth accuracy falls well below Efficient LoFTR's 56.4, the steerable backbone or the loss of absolute positional encoding is the likely cause. We would then test restoring a distance-only positional term.
References
- LoFTR: Detector-Free Local Feature Matching With Transformers
- Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed
- SRTPN: Scale and Rotation Transform Prediction Net for Multimodal Remote Sensing Image Registration
- ReF -- Rotation Equivariant Features for Local Feature Matching
- Polar Transformer Networks
- Fourier Image Transformer
- Enhancing Multi-View Smoothness for Sequential Recommendation Models
- Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture
- 3MRS: An Effective Coarse-to-Fine Matching Method for Multimodal Remote Sensing Imagery