Estimating how a camera rotated between two frames needs information from the whole image. A 90° turn moves every pixel by a large, spatially varying amount, so any network that only compares nearby pixels cannot see the match. The usual fix is to give the network a global view: dense self-attention, where every image patch compares itself with every other, or long recurrent sweeps across the image. Both are costly, and both still output rotation through an unconstrained regression head. We propose a different route. Rotation has only three degrees of freedom, and a pure camera rotation acts on the viewing sphere in the same way regardless of scene depth. So the global information a network needs for rotation can be packed into a handful of 3D vectors that rotate exactly with the camera. Our design pools rotation-equivariant features (features that transform predictably when the input rotates) into a small set of vectors per frame. It solves for rotation in closed form by aligning the two frames' vector sets, and then refines the estimate with purely local comparisons, re-centering each comparison through the rotation group's exponential map.
The payoff, if the claim holds, is a relative-rotation estimator whose accuracy stays flat across the full 0–180° range. Its cost grows linearly with image size and it has no pairwise attention. It also separates two things that current visual odometry networks mix together: the global "which way did I turn" question and the local "where exactly does this patch go" question.
Core Claim
For omnidirectional images, we claim that a global receptive field for relative rotation can be replaced by global pooling of equivariant vector features, followed by local refinement in the tangent space of $SO(3)$, the group of 3D rotations. This loses no accuracy against dense global attention and is far cheaper. The construction is simple. First, rotation-invariant scalar features $a_c(x_i)$ are computed at each token (image sample) $i$ on the sphere, with unit viewing direction $u_i$. Each frame is then summarized by $C$ vectors:
$$V_c = \sum_i a_c(x_i)\, u_i \in \mathbb{R}^3, \qquad c = 1,\dots,C.$$
When the camera rotates by $R$, the invariant scalars travel with the tokens and each $u_i$ becomes $R u_i$. Under pure rotation the second frame's vectors are therefore exactly $V'_c = R V_c$, and $R$ follows from a weighted Kabsch/SVD alignment of the two $3 \times C$ matrices. This alignment has a global receptive field at $O(N)$ cost.
Camera translation and occlusion break the exact relation, so a refinement stage follows. It compares invariant features of frame 1 at direction $u$ with frame-2 features only inside a small spherical window around $R_k u$. From the matched ray pairs it solves for a correction $\delta\theta_k \in \mathbb{R}^3$ and updates
$$R_{k+1} = \exp(\delta\theta_k^{\wedge})\, R_k,$$
where $\exp$ maps a 3-vector to a valid rotation matrix. The window is local only because $R_k$ has already removed the large, global part of the motion. Equivariance is what makes this legitimate: warping an equivariant feature map by $R_k$ gives the same result as computing features on the rotated image. With ordinary CNN features that equality fails, and warped feature maps compare mismatched quantities.
Relation to Prior Work
Two papers motivate the gap from opposite sides. Exploring Self-Attention for Visual Odometry adds a dense self-attention block over optical-flow features and trains with an $SE(3)$ Lie-algebra loss. Global context let it ignore moving cars, but its internal features are not rotation-equivariant and its attention is quadratic in image size. LARGER RECEPTIVE FIELD BASED RGB VISUAL RELOCALIZATION METHOD USING CONVOLUTIONAL NETWORK shows that widening the field of view and sweeping LSTMs across the feature map sharply cuts relocalization error. It does this by non-uniformly resizing images, which corrupts camera intrinsics, and it regresses orientation as a Euclidean vector. Our warping step requires true rays, so we keep calibrated geometry and lift pixels to directions using the known intrinsics. Dynamic Attention-based Visual Odometry widens receptive fields with dilated convolutions and also regresses rotation without manifold structure.
Two papers supply the machinery. Spherical SO(3) Equivariant Local Attention (SoLA) builds exactly $SO(3)$-equivariant attention on the sphere, with a linear-time variant. We reuse its local layers to produce the invariant scalars $a_c$. We differ in the task: SoLA does single-image recognition, where invariance is the goal. We do two-frame estimation, where we need the equivariant part and must read a rotation out of it. SO(3)-Pose shows that equivariant vector features, paired with invariant features, support 6D object pose. Its authors state that the local point-MLP backbone limits global reasoning and suggest Transformers as the fix. We test a cheaper answer to that same limitation: global equivariant pooling, not global attention.
Two further papers constrain the design. Automatic Symmetry Discovery with Lie Algebra Convolutional Network and Lie Group Algebra Convolutional Filters build equivariance from local, infinitesimal or local-kernel operators. Large transformations then require deep stacks. We avoid that by never asking a local operator to cover a large rotation. RA-NeRF shows that unconstrained pose optimization collapses under large rotations (BARF above 50° error) while flow plus 8-point/SVD geometry stays near 1°. We borrow its differentiable epipolar solve as the refinement's final step. Finally, SoLA notes that strict $SO(3)$ equivariance ignores gravity. Our Kabsch formulation admits a principled fix: when an IMU gravity vector is available, append it as an extra column of $V$. The model then stays equivariant to rotations of image and gravity jointly, but can use gravity. We treat this as a secondary experiment.
Research Program
We will build the estimator on icosphere-sampled panoramas, as SoLA does. It has three stages. The first is a shallow stack of local SoLA layers producing $C$ invariant scalar channels per token. The second is the pooled-vector Kabsch initializer, with learned per-channel confidence weights. The third is $K$ refinement iterations with a spherical correlation window of angular radius $\rho$, ending in a weighted 8-point solve on matched rays that yields rotation and translation direction. Training uses the geodesic rotation error $\|\log(R_{gt}^{\top}\hat R)\|$ at every iteration, plus an angular loss on translation direction.
The analysis asks a precise question: under what conditions is the pooled initializer inside the refinement's basin, meaning within $\rho$ of the truth? Under pure rotation the initializer is exact up to discretization. Its error should grow with the ratio of camera translation to scene depth, because parallax, not rotation, is what breaks $V' = RV$. We will measure this basin empirically as a function of that ratio. This yields a stated failure condition rather than an average score.
For perspective cameras we map pixels to a partial sphere through the known intrinsics. Equivariance then holds only for the overlapping field of view, and we expect degradation that grows with rotation magnitude. Measuring that degradation is part of the program.
Baselines
The naive reference is a CNN on the two stacked frames that regresses a rotation vector with the Lie-algebra loss. This is the AVO recipe without attention. It shows what a local receptive field does with no geometric help, and we expect it to fail once rotation exceeds what its effective receptive field covers.
The strongest global-context competitor is a dense cross-frame attention model in the style of Exploring Self-Attention for Visual Odometry. We will also run its linear-time equivariant counterpart, built from global Linear SoLA layers from Spherical SO(3) Equivariant Local Attention followed by the same Kabsch readout. The second variant isolates whether attention adds anything beyond pooling. All models share the icosphere resolution, training pairs, parameter budget within 10%, and an equal hyperparameter-search budget. Beating these baselines means matching their geodesic error within noise across all rotation magnitudes, at lower peak memory and latency on high-resolution spheres. A pure accuracy win is not required, because the claim is equivalence at lower cost.
The strongest geometric pipeline is the flow-plus-8-point/SVD approach of RA-NeRF, run on the same frame pairs. On NeRFBuster, RA-NeRF reports a mean rotation error of 1.14°, against 7.07° for CF-NeRF, and we will evaluate there using its protocol. For KITTI the file reports AVO at 0.038 deg/m rotation error and DAVO at 0.0146°/100m. We will quote these but compare only against our own reruns under a shared protocol. For relative rotation on Stanford2D3D or Structured3D panoramas, no reported result was found on Lacuna.
Three ablations isolate the key ingredients. Replacing SoLA features with an ordinary CNN on the sphere, while keeping pooling and warping identical, tests whether equivariance is what makes feature-space warping valid. Removing refinement, so the model uses the Kabsch initializer alone, tests how much the local stage contributes under parallax. Removing the initializer, so refinement starts from identity, tests whether the local window can reach large rotations on its own. We expect it cannot.
Why not just…?
Why not just use global attention and accept the cost? For small panoramas this may work as well as our method. We say so and test it directly: the global Linear SoLA baseline is our closest competitor. The argument for pooling is scaling. Linear SoLA already reports 4.1G FLOPS against 5.9G for a non-equivariant baseline on Spherical MNIST, but it still performs a full attention pass per layer, whereas our global step is one weighted sum and a 3×3 SVD. If global attention matches our accuracy at comparable memory on high-resolution spheres, the efficiency claim fails. The remaining contribution would then be the analysis of when rotation is recoverable from pooled vectors.
Why not just use an image pyramid, coarse to fine? Pyramids enlarge reach by downsampling, which is the resolution loss that CCNet and ConvFormer describe as the cost of locality. A pyramid of non-equivariant features also cannot handle large rolls about the viewing axis, where displacement varies in direction across the image. This alternative is a genuine threat for moderate rotations. We will include a pyramid local-correlation baseline and report error against rotation magnitude and axis.
Why not just read rotation from a gyroscope? On an inertial-equipped robot, short-term gyro integration gives relative rotation cheaply, which is a real limit on our method's value there. Gyro bias still causes drift over long horizons, as Neural Inertial Odometry from Lie Events discusses, and many cameras have no IMU. On EuRoC, which has both, we will run gyro-only, ours-only, and ours initialized from the gyro. The question is whether vision corrects gyro drift at low cost. We do not claim to replace the gyro.
Why not just augment a standard regressor with random rotations? Existing evidence argues against this. In Leveraging SO(3)-steerable convolutions for pose-robust semantic segmentation in 3D medical data, nnUnet with augmentation reached Dice 0.76 against 0.85 for the steerable model. On Stanford2D3D, SphereUFormer fell from 72.2 to 15.3 mIoU under rotation while SoLA held 72.2. Our naive baseline is trained with full $SO(3)$ augmentation, so it measures this directly.
Decisive Experiments
The central test uses Stanford2D3D and Structured3D panoramas. Synthetic pure rotations drawn uniformly over $SO(3)$ give exact ground truth. Real pose pairs, where the datasets provide camera positions, introduce parallax. We plot geodesic error against rotation magnitude from 0° to 180°. The prediction is that our curve stays flat and matches global attention, while the naive and no-initializer variants fail past roughly the window size. We then plot peak GPU memory and latency against icosphere resolution; the prediction is linear growth for ours and a steeper slope for attention.
Next we plot initializer error against the ratio of camera translation to scene depth, which locates the basin boundary where refinement stops rescuing the estimate. The non-equivariant ablation should fail under large rotations even with identical pooling. If it does not, equivariance is not the reason the method works, and we will say so.
Transfer tests run on KITTI and TUM RGB-D with rays from calibrated intrinsics, reporting relative rotation error, and on NeRFBuster with RA-NeRF's rotation-error protocol. The gravity-column variant is evaluated on EuRoC, where roll and pitch error should drop relative to strictly equivariant pooling. Reaching the RA-NeRF regime of roughly 1° on NeRFBuster without incremental registration would be strong support for the approach. Matching global attention on panoramas at a fraction of its memory would be the minimal success.
References
- Exploring Self-Attention for Visual Odometry
- LARGER RECEPTIVE FIELD BASED RGB VISUAL RELOCALIZATION METHOD USING CONVOLUTIONAL NETWORK
- Dynamic Attention-based Visual Odometry
- Spherical SO(3) Equivariant Local Attention
- SO(3)-Pose: SO(3)-Equivariance Learning for 6D Object Pose Estimation
- Automatic Symmetry Discovery with Lie Algebra Convolutional Network
- Lie Group Algebra Convolutional Filters
- RA-NeRF: Robust Neural Radiance Field Reconstruction with Accurate Camera Pose Estimation under Complex Trajectories
- CCNet: Criss-Cross Attention for Semantic Segmentation
- ConvFormer: Combining CNN and Transformer for Medical Image Segmentation
- Neural Inertial Odometry from Lie Events
- Leveraging SO(3)-steerable convolutions for pose-robust semantic segmentation in 3D medical data