Introduction
Remote sensing (RS) imagery presents a unique challenge due to its multi-scale nature. Hardware constraints often force a trade-off between spatial resolution and spectral or temporal coverage, resulting in datasets with varying Ground Sample Distances (GSD). This variability leads to two primary hurdles: the scarcity of labeled data across all scales and the difficulty of aligning features from images captured at different resolutions. Conventional models often ignore these spatial scale characteristics, leading to sub-optimal feature extraction. To address this, the authors propose Cross-Scale MAE, a self-supervised framework designed to learn scale-invariant features without requiring pre-aligned multi-scale pairs.
Methodology
The Cross-Scale MAE framework is built upon the Masked Auto-Encoder (MAE) architecture, which learns by reconstructing masked image patches. The model introduces three key innovations to handle multi-resolution data:
- Scale Augmentation: Instead of requiring real-world aligned image pairs, the framework synthetically generates two versions of a single image at different scales through random cropping and resizing. This forces the model to learn features that persist across different "zoom levels."
- Cross-Scale Consistency (Encoder): A contrastive loss (InfoNCE) is applied at the encoder level. It pushes the representations of the same site at different scales closer together in the feature space while pushing representations of different sites further apart. This ensures structural consistency, allowing the model to identify the same objects regardless of resolution.
- Cross-Scale Prediction (Decoder): The decoder implements a cross-prediction mechanism where embeddings from a low-resolution view are used to predict the embeddings of the high-resolution version. This ensures semantic consistency, helping the model understand what those objects are across resolutions.
To make the model accessible for researchers with limited resources, the authors leveraged the xFormers library, optimizing memory and speed to allow pre-training on a single GPU without sacrificing performance.
Experimental Evaluation
The model was pre-trained on the fMoW-RGB (Functional Map of the World) dataset, containing over 363k images with varying resolutions. Evaluation was conducted on several standard benchmarks: RESISC45, WHU-RS19, UC Merced, and EuroSAT for classification, and Potsdam and Vaihingen for segmentation.
Classification Performance and Robustness
Cross-Scale MAE consistently outperformed baselines such as SatMAE and the previous state-of-the-art, Scale-MAE. On the RESISC45 dataset, it achieved a KNN classification accuracy of 75.6%, significantly higher than SatMAE (66.3%) and Scale-MAE (70.0%).
A key finding of the study is the model's robustness to extreme scaling. Even when images were downscaled to 12.5% of their original size—a range the model did not encounter during training—Cross-Scale MAE maintained a significant performance lead over its peers.
Downstream Segmentation
In semantic segmentation tasks on the Potsdam dataset, the model achieved a Mean Intersection over Union (mIoU) of 76.17%, outperforming Scale-MAE (75.07%). This demonstrates that the scale-invariant features learned during pre-training translate effectively to dense prediction tasks.
Discussion and Conclusion
The success of Cross-Scale MAE suggests that simulated multi-scale augmentation is a highly effective substitute for expensive, perfectly aligned multi-sensor data. One of the most powerful insights is that simple resizing combined with the right loss functions can bridge the GSD gap.
While highly effective for spatial scaling, the authors acknowledge limitations regarding spectral and temporal gaps. Future work aims to incorporate a more holistic multi-spectral perspective and investigate more refined contrastive strategies to further improve the model's ability to handle the diverse nature of satellite imagery.
Summary updated