Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data

Oscar Mañas, Alexandre Lacoste, Xavier Giró-i-Nieto, David Vázquez, Pau Rodríguez

Abstract

Remote sensing and automatic earth monitoring are key to solve global-scale challenges such as disaster prevention, land use monitoring, or tackling climate change. Although there exist vast amounts of remote sensing data, most of it remains unlabeled and thus inaccessible for supervised learning algorithms. Transfer learning approaches can reduce the data requirements of deep learning algorithms. However, most of these methods are pre-trained on ImageNet and their generalization to remote sensing imagery is not guaranteed due to the domain gap. In this work, we propose Seasonal Contrast (SeCo), an effective pipeline to leverage unlabeled data for in-domain pre-training of remote sensing representations. The SeCo pipeline is composed of two parts. First, a principled procedure to gather large-scale, unlabeled and uncurated remote sensing datasets containing images from multiple Earth locations at different timestamps. Second, a self-supervised algorithm that takes advantage of time and position invariance to learn transferable representations for remote sensing applications. We empirically show that models trained with SeCo achieve better performance than their ImageNet pre-trained counterparts and state-of-the-art self-supervised learning methods on multiple downstream tasks. The datasets and models in SeCo will be made public to facilitate transfer learning and enable rapid progress in remote sensing applications.

Introduction

Deep learning has revolutionized remote sensing (RS), but it faces a massive bottleneck: data labeling. While satellites generate terabytes of imagery daily, annotating this data requires expert knowledge and is prohibitively expensive. To bypass this, researchers usually use "transfer learning," taking a model pre-trained on ImageNet (natural photos of cats, cars, etc.) and fine-tuning it on satellite data. However, there is a significant domain gap between a handheld camera photo and a top-down multispectral satellite tile. Furthermore, existing RS datasets are often "highly curated"—hand-selected to be clean and balanced—which doesn't reflect the messy, "uncurated" reality of global satellite streams.

Methodology

The authors propose a two-part solution to bridge this gap: a new uncurated dataset and the Seasonal Contrast (SeCo) algorithm.

Uncurated Data Collection

The researchers developed a principled way to gather a massive, unlabeled dataset from Sentinel-2 satellites. Instead of random sampling (which yields mostly empty ocean or desert), they sampled around 10,000 populated cities to ensure geographic and structural diversity. For each location, they pulled images from five different timestamps across a year. The resulting SeCo dataset contains approximately 1 million multi-spectral image patches from 200,000 locations worldwide.

The SeCo Algorithm

SeCo is a self-supervised algorithm based on contrastive learning (specifically MoCo-v2). Standard contrastive learning takes one image and creates two "views" using artificial augmentations (like cropping or flipping) and tells the model these should have similar representations.

The SeCo architecture uses multiple heads to learn representations invariant to seasonal changes and artificial augmentations.

SeCo introduces "seasonal positive pairs," treating images of the same location at different times of the year as versions of the same thing. This leverages natural variations—such as snow cover or vegetation cycles—as a form of "free" supervision. To handle different downstream requirements, SeCo uses a multi-head architecture to project representations into three sub-spaces: one invariant to everything, one variant to seasonal changes, and one variant to artificial augmentations. This disentanglement allows the model to encode both what changes (useful for change detection) and what stays the same (useful for land-cover classification).

Experimental Results

SeCo was evaluated on several benchmarks, including BigEarthNet (land-cover classification) and OSCD (change detection), using both linear probing and fine-tuning.

Performance and Label Efficiency

On BigEarthNet, SeCo achieved a Mean Average Precision (mAP) of 87.81%, significantly higher than ImageNet pre-training (86.74%) and standard MoCo-v2 (86.05%). The most striking result is the model's label efficiency: a SeCo model trained with only 1% of the labels outperformed an ImageNet-initialized model trained on 100% of the labels.

Comparison of SeCo against baselines across different percentages of labeled data, demonstrating superior label efficiency.

Change Detection

On the OSCD task, SeCo achieved an F1 score of 46.94, outperforming the next best self-supervised baseline (MoCo-v2) by over 6 points. This highlights the benefit of the multi-head architecture in capturing temporal variances that are often discarded by standard contrastive methods.

Conclusion

Seasonal Contrast demonstrates that the temporal dimension of satellite imagery provides a powerful source of supervision. By leveraging uncurated data and seasonal variations, SeCo produces robust, transferable representations that significantly reduce the need for expensive manual labeling in remote sensing applications. While challenges like cloud cover and full spectral integration remain, SeCo provides a scalable path forward for global-scale earth monitoring.

Summary updated

Abstract

Remote sensing and automatic earth monitoring are key to solve global-scale challenges such as disaster prevention, land use monitoring, or tackling climate change. Although there exist vast amounts of remote sensing data, most of it remains unlabeled and thus inaccessible for supervised learning algorithms. Transfer learning approaches can reduce the data requirements of deep learning algorithms. However, most of these methods are pre-trained on ImageNet and their generalization to remote sensing imagery is not guaranteed due to the domain gap. In this work, we propose Seasonal Contrast (SeCo), an effective pipeline to leverage unlabeled data for in-domain pre-training of remote sensing representations. The SeCo pipeline is composed of two parts. First, a principled procedure to gather large-scale, unlabeled and uncurated remote sensing datasets containing images from multiple Earth locations at different timestamps. Second, a self-supervised algorithm that takes advantage of time and position invariance to learn transferable representations for remote sensing applications. We empirically show that models trained with SeCo achieve better performance than their ImageNet pre-trained counterparts and state-of-the-art self-supervised learning methods on multiple downstream tasks. The datasets and models in SeCo will be made public to facilitate transfer learning and enable rapid progress in remote sensing applications.
BibTeX
@inproceedings{manas2021seasonal,
  author = {Oscar Mañas and Alexandre Lacoste and Xavier Giró-i-Nieto and David Vázquez and Pau Rodríguez},
  title = {{Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00928},
}
Figure