Merging Tokens Without Losing the Geometry: Calibration-Free Cross-View Localization

Cross-view localization means finding where a car or pedestrian camera is on a satellite map. The best current method matches small image patches between the ground photo and the aerial tile. It then lifts the ground matches into 3D using a predicted depth map and solves for the camera's map position in closed form. This works well, but it has two costs. It must know the camera's focal length and optical center (its "intrinsics") in order to turn pixels into rays. It also compares every ground patch against every aerial patch, so the cost grows quickly with resolution and search area. Standard ways of speeding up Vision Transformers merge similar patch tokens into averaged tokens. Doing that naively here throws away the pixel positions the geometry depends on.

This note argues that the two problems share one fix. For an upright camera, the position of a point on the ground plane depends linearly on two per-pixel quantities: depth times horizontal pixel coordinate, and depth. Two consequences follow. Unknown horizontal intrinsics can be absorbed into a plain 2D affine fit instead of a rigid alignment. And a merged token that stores the average and spread of those two quantities is exact for that fit, up to an uncertainty it can report itself. The result is a token-merging rule designed for geometry. The merged tokens make localization cheaper, the localization needs no camera calibration, and the merge criterion and the weights in the pose solver both come from the same linear model.

Core Claim

The method builds on Loc². Loc² matches frozen DINOv2 patch features between a ground image and an aerial tile, lifts ground matches to 3D with monocular depth, and solves a differentiable scale-aware Procrustes alignment (rotation, translation, scale). That lifting needs known intrinsics and known vertical orientation, and it runs on full, uncompressed token maps.

We keep the gravity assumption, which is reasonable for vehicle cameras, and drop the intrinsics. A pixel with horizontal coordinate $u$ and depth $d$ has camera-frame ground coordinates $(d(u-c_x)/f,\ d)$, where $f$ is the focal length and $c_x$ is the principal point. With $a = 1/f$ and $b = -c_x/f$, the map position is

$$ p_{\text{map}} = M\,\phi + t, \qquad \phi = (d\,u,\ d), \qquad M = s\,R\begin{pmatrix} a & b \\ 0 & 1\end{pmatrix}, $$

where $s$ is the unknown depth scale, $R$ is the in-plane rotation (heading), and $t$ is the map translation. $M$ is a general 2×2 matrix and $t$ a 2-vector, so the problem is an unconstrained weighted affine least-squares fit with six unknowns. It has a closed-form solution. An RQ decomposition of the fitted $M$ then returns heading, scale, focal length and principal point.

The same linearity governs merging. If a merged token stores the weighted mean $\bar\phi_c$ of its pixels' $\phi$, the cluster's contribution to the fit equals the sum of its pixels' contributions, minus a term set by the within-cluster covariance $\Sigma_c$ of $\phi$. A cluster with small $\Sigma_c$ loses nothing. A cluster that straddles a depth discontinuity, such as a lamppost against a far facade, has large $\Sigma_c$. It corresponds to an extended region on the map rather than one location, and it should either not be formed or be down-weighted. We therefore merge with the cost

$$ \text{cost}(i,j) = \big(1 - \cos(k_i, k_j)\big) + \lambda\, \frac{\operatorname{tr}\Sigma_{i\cup j}}{\bar d_{i\cup j}^{\,2}}, $$

where $k$ are attention keys (the similarity Agglomerative Token Clustering uses), $\Sigma_{i\cup j}$ is the covariance of the merged cluster, $\bar d$ is its mean depth (which makes the penalty scale-free), and $\lambda$ trades appearance against geometry. The affine solver weights each match by match confidence times pixel count, divided by $\operatorname{tr}\Sigma_c$. Aerial tokens are merged in the same way, carrying first and second moments of their map coordinates. One caveat: the derivation needs depth that is correct up to scale only, not up to scale plus shift. We therefore use scale-only depth predictors and test this assumption explicitly.

Relation to Prior Work

Several papers motivate the gap. EViT fuses low-attention tokens into one summary token. ATC clusters tokens bottom-up on key similarity and beats bipartite merging by 4 mAP on COCO detection at a 50% keep rate. Both show that aggregation beats deletion, but neither carries spatial coordinates. They have been tested only on single-view recognition and detection, never on tasks where merged tokens must be projected into 3D.

Two cross-view methods group tokens without reducing compute or preserving metric geometry. FSRA partitions patch tokens into heat-ranked regions for drone-to-satellite retrieval. InfoGeo routes tokens into concept slots. EgoTR removes hand-coded geometric priors by learning positional embeddings, but it outputs a global retrieval descriptor rather than a metric pose and keeps all tokens through 12 layers. EMIFF does compress features across steep viewpoints, but it conditions channel masks and voxel projection on known calibration. That is the dependency we remove.

Two papers supply method. AnyCalib recovers intrinsics from a single image through regressed field-of-view fields. We use it as the strongest "calibrate first" competitor and as a cross-check on the focal lengths our affine fit recovers. EpipolarPose showed that multi-view geometry can replace extrinsic calibration as supervision. We take the same stance for intrinsics, using the map as the geometric constraint.

Research Program

The work has three parts.

The first part is analytic and synthetic. We derive the exact decomposition of the merged-cluster least-squares objective into a centroid term and a covariance term. We check it on rendered scenes with known geometry: an unmerged fit and a moment-merged fit must agree whenever clusters have zero lifted covariance. We also characterize degeneracies, which mirror the epipolar degeneracies listed for uncalibrated geometry. When matches lie at nearly constant depth or in a narrow horizontal band, $a$ and $b$ become unidentifiable, and the solver should then fall back to a prior focal length.

The second part builds the system. It consists of Loc²'s frozen DINOv2 features and projection head, a scale-only monocular depth model, GPU-batched agglomerative merging with the cost above (ATC's CPU-bound clustering is a known latency risk), and the affine solver, wrapped in a RANSAC-style hypothesis loop for unknown heading. The projection head is fine-tuned through the differentiable solver.

The third part is an evaluation protocol for unknown intrinsics. On KITTI's cross-view split we randomly crop and rescale each ground image, which changes both $f$ and $c_x$, and hide the calibration from every method except an oracle. On VIGOR we cut perspective views of random field of view and heading from the panoramas.

Baselines

The oracle reference is Loc² with true intrinsics and all tokens, retrained under our crop augmentation. Its published numbers set the scale: a mean error of 5.60 m on KITTI cross-area against a previous best of 7.31 m, a mean error of 4.23 m on VIGOR with unknown orientation against 10.02 m before, and 14.74 FPS on an H100, all as reported in Loc². The original paper did not use our cropped protocol. We therefore rerun it under identical data, the same backbone, and the same hyperparameter search budget, and report both the published and the rerun figures.

The naive reference is Loc² fed a fixed nominal focal length and centered principal point under the crop protocol. This measures how much miscalibration costs, which is the floor any calibration-free method must clear.

The strongest calibration-free competitor is AnyCalib followed by full-token Loc². AnyCalib runs at roughly 25 ms on an RTX 4090. No per-dataset angular-error numbers were found in the research file, and no reported result found on Lacuna for AnyCalib on KITTI or VIGOR cross-view localization. We run its released model unchanged and count its latency in the total. Our affine solver beats it if it matches or improves localization error at equal or lower total latency. If AnyCalib plus Loc² wins on error, the solver component should be dropped in favor of AnyCalib and the contribution narrows to merging.

For efficiency, the strongest competitor is ATC applied to Loc²'s ground and aerial tokens. Each merged token is assigned only its pixel centroid, with depth sampled at that centroid. It is given the same token budgets (50%, 25% and 10% keep rates). There is no reported result found on Lacuna for ATC on cross-view localization, and the closest figure is its 4 mAP gain over ToMe on COCO. Beating it means lower median error at every matched budget, with the gap growing as the keep rate falls.

Three ablations isolate the ingredients: the same merging with key similarity only (no depth gating, $\lambda = 0$), storing $\bar\phi$ but using uniform weights (no covariance weighting), and centroid-depth lifting instead of moment-exact lifting.

Why not just…?

Why not just lower the input resolution or use larger patches? This is the most serious threat. A coarser regular grid keeps exact coordinates and needs no clustering. Our argument is that content-adaptive merging spends its budget on curbs, building corners and pole bases, the structures that pin down the map pose, and collapses sky and road interiors. Uniform downsampling blurs small landmarks everywhere. That is an empirical claim. We compare error against FLOPs, matched exactly, for 14- versus 28-pixel-stride inputs and for our merging. If the coarse grid matches our curve, the merging contribution fails and we will report that.

Why not just prune the low-confidence tokens? Loc² already weights matches by confidence, so dropping weak matches might seem free. Evidence from EViT and ATC that deletion loses context comes from recognition tasks, not from pose solvers. Here the open question is whether tokens from the road surface and the background tighten the affine fit by spreading the depth and lateral support that identifies $a$ and $b$. We test top-$k$ pruning at matched budgets and track focal-recovery error separately. We expect pruning to hurt intrinsics recovery more than position.

Why not just calibrate with AnyCalib first and keep Loc² as is? This works on any camera model, including distortion that our pinhole-horizontal derivation ignores. It is a real alternative and is run as a full baseline. The affine solver's advantages would be that it is jointly consistent with the map, that it adds no network pass, and that it is unaffected by the crop and editing degradations known to make single-image principal-point estimation ill-posed. If those advantages do not show up as lower error, AnyCalib becomes the calibration module and only the merging is kept.

Why not just warp the ground image to a bird's-eye view? Bird's-eye warping needs intrinsics in the first place. Loc² also showed that matching in the image plane beats warping: its error drops from 10.02 m to 4.23 m on VIGOR with unknown orientation. We do not revisit that comparison.

Decisive Checks

Four results would decide the project.

The first is the synthetic exactness test. Moment-merged and unmerged affine solutions must coincide when within-cluster covariance is zero, and diverge in proportion to $\operatorname{tr}\Sigma_c$ otherwise.

The second is the main KITTI and VIGOR crop-protocol result. At a 25% token budget, our system should keep median error within the run-to-run noise of full-token oracle Loc², while centroid-only ATC and confidence pruning degrade measurably. Run-to-run noise is measured over seeds.

The third is focal-length and principal-point recovery error from the RQ decomposition, compared against AnyCalib on the same crops, broken down by how much depth spread the matches have.

The fourth is the depth-shift test. Replacing scale-only depth with affine-invariant depth should break the method in a predictable way. That would confirm the linear model is doing the work rather than the learned head.

Tilted and oblique aerial imagery, and arbitrary camera roll, are outside the scope of this proposal and stated as limitations.