This research, presented at AAAI-25, introduces a refined self-supervised learning approach for hyperspectral satellite imagery. By replacing the standard "random masking" in Masked Autoencoders (MAE) with a strategy driven by the Histogram of Oriented Gradients (HOG), the authors demonstrate significant improvements in image reconstruction and feature learning, particularly in geographically unseen regions.
The Problem: Sub-optimal Random Masking
Masked Autoencoders (MAE) typically rely on random masking, where a large portion of an image (often 75%) is hidden, and the model is tasked with reconstructing the missing patches. While effective for standard datasets like ImageNet, random masking is often sub-optimal for hyperspectral satellite imagery. Satellite data is dense and spatially diverse, containing complex topographical features across hundreds of spectral bands. Randomly masking these images can result in the loss of critical structural information, such as edges and textures, which are vital for learning meaningful geospatial representations.
The HOGMAE Approach
To address this, the authors propose HOGMAE, a "zero-shot" masking strategy that identifies important spatial features before training begins. Instead of expensive pre-trained attention mechanisms, HOGMAE utilizes the Histogram of Oriented Gradients (HOG)—a classic computer vision descriptor—to guide the masking process.
The HOGMAE workflow involves:
- Dimensionality Reduction: Using Principal Component Analysis (PCA) to identify the single channel with the highest variance from the hundreds of hyperspectral bands.
- Feature Extraction: Computing HOG features for every image patch. Patches with high HOG values represent high visual information (e.g., sharp edges or complex textures).
- Biased Masking: Patches are categorized by variability. The strategy is biased to retain high-variability patches for the encoder while masking more of the low-variability (flatter) patches.
- Reconstruction: The model is trained to reconstruct the masked patches using a standard MAE architecture with a 75% total masking ratio.
Experimental Results
The study utilized data from the Environmental Mapping and Analysis Program (EnMAP), consisting of over 22,000 tiles with 218 spectral bands. The model was evaluated on two test sets, with Test Dataset 2 representing "unseen" geospatial areas to test generalization.
The HOG-based masking strategy consistently outperformed random masking across all metrics. On the unseen test set, HOGMAE achieved a 17.6% reduction in Mean Absolute Error (MAE), a 5.4% increase in PSNR, and a 4.8% increase in SSIM.
Key Insights and Future Work
The results suggest that for satellite imagery, the spatial location of the mask is as critical as the masking ratio. By ensuring the encoder sees the "skeleton" of the landscape (the edges and shapes), the decoder becomes significantly more adept at filling in the gaps. This approach is particularly effective for generalizing to new terrains, as it helps the model learn universal topographical rules.
Future work will explore the computational overhead of the HOG/PCA preprocessing and evaluate the performance of HOGMAE-trained models on downstream tasks such as land-cover classification and change detection.
Summary updated