Satellite, aerial and underwater survey images come with free structure: every tile has a known location, and nearby tiles tend to look alike. Spatial contrastive learning uses this. It pretrains an image encoder without labels by pulling the embeddings of nearby tiles together and pushing distant ones apart, then fits a small classifier on a few labeled examples. Two problems remain. First, the objective contradicts itself. Batches routinely contain tiles from the anchor's own neighborhood, and the loss is told to push those apart even though the location prior says they should be close. Current fixes simply drop nearby tiles from the set of negatives. Second, top-down imagery has no canonical "up." A field photographed on a north-south flight and one photographed on an east-west flight are the same field, yet standard encoders treat them as different inputs.
We propose one change that addresses both problems. The encoder is not asked to make neighbors similar. It is asked to predict the neighbor's embedding from the anchor's embedding and the known displacement vector between the two tiles. That prediction is built to be equivariant to rotations and reflections: rotating the whole scene rotates the displacement and transforms the prediction in a matching, known way. A nearby negative no longer conflicts with the objective. It is simply the tile at a different offset. The embedding separates into a part that ignores orientation (what the place is) and a part that tracks it (which way crop rows, ridges or shadows run). The payoff we test is label efficiency: better accuracy with 1–10% of the labels, and better choice of which few tiles to send to an expert.
Core Claim
Treating the spatial neighbor as a transformed view whose transformation is known, namely a translation by $\delta$ composed with an optional rotation or reflection $g$, gives a spatial pretraining signal that is both conflict-free and symmetry-aware. We claim three things. This beats spatial contrastive baselines that rely on exclusion buffers and rotation augmentation in the low-label regime. The gain comes from the displacement-conditioned prediction rather than from the equivariant architecture alone. The orientation-free part of the embedding is a better space for choosing which tiles to label.
Concretely, the encoder $f$ maps a tile $x$ to $z = (u, v)$. Here $u$ is unchanged under the dihedral group $D_4$ (the eight 90° rotations and flips, which act exactly on pixel grids), and $v$ transforms by a fixed matrix $\rho(g)$ when the input is transformed by $g \in D_4$. A predictor $h$ takes the anchor embedding $z_a$ and the displacement $\delta$ to the neighbor and outputs a guess for the neighbor's embedding $z_n$. It is trained with a contrastive loss over a batch of candidate tiles $\{z_m\}$:
$$ \mathcal{L} = -\log \frac{\exp\big(\mathrm{sim}(h(z_a,\delta), z_n)/\tau\big)}{\sum_m \exp\big(\mathrm{sim}(h(z_a,\delta), z_m)/\tau\big)}, $$
where $\mathrm{sim}$ is cosine similarity and $\tau$ a temperature. The key constraint is built into the architecture rather than added as a penalty:
$$ h\big((u, \rho(g)v),\, g\delta\big) = \rho(g)\, h\big((u,v),\delta\big) \quad \text{for all } g \in D_4 . $$
In words, rotating the scene and rotating the displacement rotates the prediction. The backbone is a $D_4$ group-convolutional ResNet. The predictor treats $\delta$ as a vector feature in the same representation as $v$ and mixes them with equivariant layers.
Relation to Prior Work
Three spatial contrastive methods motivate the proposal and supply its setting. Tile2Vec introduced the anchor–neighbor–distant triplet loss on geographic tiles and showed that features from a Random Forest on top of it beat ImageNet features for cropland classification. Accelerating Ecological Sciences from Above applied a softmax spatial contrastive loss to invasive-species habitat prediction and used the embedding for active field surveys. OmniSat aligned multiple sensors by location and explicitly removed geographically adjacent patches from the negative pool. All three use orientation-blind backbones. All three either ignore the neighbor–negative conflict or handle it with a buffer. None uses the displacement vector itself, which is exactly the information we condition on. The same conflict appears in spatio-temporal forecasting (Beyond Fixed Variables), where it is also handled by hand-built exclusion rules. MetaStreet borrows negatives from labeled graph neighbors. That is a different fix, tied to street graphs, and it does not remove the conflict for raster tiles.
Two equivariant self-supervised methods supply the idea of splitting the embedding, but not the spatial signal. Equivariant Contrastive Learning adds a rotation-prediction head beside an invariant contrastive branch and gains about a point on ImageNet. CARE forces augmentations to act as orthogonal rotations in latent space. Both operate on synthetic augmentations of independent natural images. Here the transformation is a physical relation between two different real tiles (their offset), and the symmetry is exact in the architecture rather than encouraged by a loss.
The case that symmetry reduces label needs comes mostly from molecules. NequIP matched a descriptor-based model with 133 rather than 133,500 training structures. SYMGP showed that symmetry-averaged kernels cut labels about 5× on QM9 and improve calibration. These results motivate the direction but constrain our expectations. Their baselines are invariant descriptors, not rotation-augmented image networks, and pixel grids with random 90° flips are a much easier case for augmentation. We therefore expect gains of a few points in the low-label regime, not orders of magnitude, and design the experiments so that a null result is informative.
The label-selection component builds on Guiding Labelling Effort for Efficient Learning With Georeferenced Images (LGA-SS). It clusters a location-regularized autoencoder's latent space and sends cluster centers to experts. With 40 such labels it reached 85% of the accuracy of 10,000 random labels on seafloor imagery. We reuse its protocol unchanged and only swap in our orientation-free embedding $u$. Rotated copies of the same substrate should then fall into one cluster rather than several, so a fixed budget covers more distinct habitats.
Research Program
The work has three stages.
The first stage builds the encoder and predictor. Pretraining samples an anchor tile, a neighbor at a random offset within radius $r$, and a random $g \in D_4$ applied consistently to both tiles and to $\delta$. Because the loss rewards predicting a specific offset, it should also be less sensitive to the choice of $r$ than neighbor-similarity losses. We test this directly by sweeping $r$.
The second stage is analysis. We check that the invariant part $u$ is truly unchanged under $D_4$, which holds by construction. We then measure how much it drifts under non-grid rotations such as 30° or 45°, which only interpolation can produce and which real flight headings create. We also measure whether $v$ encodes offset and orientation, by retrieving the true neighbor among distractors given an anchor and $\delta$.
The third stage evaluates label efficiency. We train linear and Random Forest probes on $u$ alone and on $(u,v)$ at 1%, 5%, 10% and 100% of labels. We also run the LGA-SS centroid-selection protocol in each candidate embedding.
Pretraining is unimodal, on single-sensor imagery, which OmniSat leaves unaddressed. Group convolutions multiply per-filter cost. We therefore match every compared model on both parameter count and training FLOPs, reducing channel width for the equivariant model as needed. The constraint-relaxation code in equivariant_optimization_cr gives a fallback if exact equivariance proves hard to optimize.
Baselines
The naive reference points are a randomly initialized backbone of matched size and an ImageNet-pretrained ResNet, each trained or probed on the same labeled subsets. They show whether any pretraining matters at 1–10% labels. On California cropland, Tile2Vec reports 47.7% accuracy for ImageNet ResNet features with a Random Forest, against 52.6% for Tile2Vec.
The strongest existing competitor is a spatial contrastive encoder with every cheap fix applied. It combines the softmax spatial contrastive loss of Accelerating Ecological Sciences from Above, the adjacent-negative exclusion buffer of OmniSat, and full random $D_4$ plus arbitrary-angle rotation augmentation. It runs on the same backbone family, pretraining tiles, epochs, FLOPs, and hyperparameter search budget as our method (the same number of trials over learning rate, temperature and radius). The reported numbers this baseline must beat are:
- 52.6% for Tile2Vec on the cropland data;
- 25.63% top-1 across 10 invasive species on the New York dataset, reported by Accelerating Ecological Sciences from Above;
- on seafloor imagery, LGA-SS's 85% of full-supervision accuracy with 40 prioritized labels.
We found no reported result on Lacuna for spatial contrastive pretraining at 1–10% labels on EuroSAT or BigEarthNet. The only figures there are full-label ones: 98.57% for a fine-tuned ResNet-50 on EuroSAT and 69.93% precision for S-CNN-All on BigEarthNet, both from the benchmark papers (EuroSAT, BigEarthNet). We will rerun all baselines ourselves at those label fractions. Beating the strongest baseline means a gain larger than the seed-to-seed spread over at least five pretraining seeds, at the 1% and 5% fractions, on at least three of the four datasets.
Three ablations isolate the key ingredients:
- Equivariant backbone with the ordinary neighbor-similarity loss. This tests the architecture alone.
- Displacement-conditioned predictor on a standard backbone with rotation augmentation. This tests the prediction target alone.
- Our full model with $v$ discarded at probe time. This tests whether orientation information helps.
A fourth control replaces the exclusion buffer with negatives down-weighted smoothly by geographic distance. This is the obvious soft version of existing practice.
Why not just…?
Why not just add random rotations and flips to Tile2Vec? This is the most serious threat. On pixel grids, $D_4$ augmentation is nearly free, and the molecular evidence from NequIP compares against invariant descriptors, not augmentation, so it does not settle the question for images. Augmentation, however, makes the embedding forget orientation entirely. Equivariant Contrastive Learning and CARE both found that forcing invariance to transformations discards useful features. The second ablation is our fair test: displacement prediction on an ordinary backbone with augmentation. If it matches the full model, we will report that the equivariant architecture is not worth its cost, and the contribution reduces to the displacement-conditioned objective.
Why not just keep the exclusion buffer, or down-weight nearby negatives by distance? Both are cheaper than our predictor. Both still ask neighbors to be similar, which blurs genuine boundaries such as a coastline or a forest edge inside radius $r$. Conditioning on $\delta$ lets the model predict that the tile to the east is water. The distance-weighted control and the radius sweep settle this. If soft weighting matches our method and is equally insensitive to $r$, the conditioning is not the active ingredient.
Why not just fine-tune an ImageNet model? The file's evidence argues against it in this domain. On BigEarthNet, an ImageNet Inception-v2 reached 48.23% precision against 65.06% for a small network trained from scratch on RGB (BigEarthNet). On the cropland data, Tile2Vec features beat ImageNet features. We still include ImageNet fine-tuning as a naive baseline at every label fraction.
Why not just label randomly? LGA-SS showed that random sampling missed rare seafloor classes that centroid selection found with 20 labels (LGA-SS). Our question is narrower: does selecting in the orientation-free space $u$ improve on selecting in an orientation-blind space? We test it with identical budgets.
Proposed Experiments
Pretraining uses unlabeled NAIP tiles for the California cropland task and Sentinel-2 tiles for EuroSAT and BigEarthNet. Where the LGA-SS benthic survey data are available, we also use seafloor mosaics. That setting is the purest test, since an underwater vehicle has no preferred heading. If those data are not available, the New York invasive-species task stands in as the rare-class test.
The main result is a set of label-efficiency curves: accuracy for EuroSAT, cropland and invasive species, precision for BigEarthNet, and accuracy for benthic substrates, each at 1%, 5%, 10% and 100% labels, with five seeds per point. The labeled subsets are fixed across all methods.
Three decisive checks follow.
- Arbitrary-angle rotation test. We rotate test tiles by non-grid angles (15°, 30°, 45°) and measure the accuracy drop. We expect our model's drop to be smaller than the augmented baseline's. If it is not, the "no canonical up" argument fails in practice.
- Neighbor retrieval test. Given an anchor and $\delta$, we retrieve the true neighbor from 100 candidates that include same-neighborhood distractors. This directly measures whether the conflict is resolved. The buffer baseline cannot condition on $\delta$, so we compare it by nearest-embedding retrieval.
- Selection test. We run LGA-SS centroid selection at budgets of 20, 40 and 100 labels in each embedding and score both overall accuracy and recall on the rarest classes.
The proposal is falsified in its strong form if the augmented displacement-prediction ablation matches the full model on all four datasets. It is falsified entirely if the fully tuned buffer-plus-augmentation baseline matches both.
References
- Tile2Vec: Unsupervised Representation Learning for Spatially Distributed Data
- Accelerating Ecological Sciences from Above: Spatial Contrastive Learning for Remote Sensing
- OmniSat: Self-supervised Modality Fusion for Earth Observation
- Beyond Fixed Variables: Expanding-variate Time Series Forecasting via Flat Scheme and Spatio-temporal Focal Learning
- MetaStreet: Semi-Supervised Multimodal Learning for Street-Level Socioeconomic Prediction
- Equivariant Self-Supervised Learning: Encouraging Equivariance in Representations
- Structuring Representation Geometry with Rotationally Equivariant Contrastive Learning
- E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials
- Symmetry-Constrained Gaussian Processes for Sample-Efficient Molecular Property Prediction
- Guiding Labelling Effort for Efficient Learning With Georeferenced Images
- EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
- BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding