The color out of space: learning self-supervised representations for Earth Observation imagery

Stefano Vincenzi, Angelo Porrello, Pietro Buzzega, Marco Cipriano, Pietro Fronte, Roberto Cuccu, Carla Ippoliti, Annamaria Conte, Simone Calderara

TL;DR

This paper introduces a self-supervised learning approach for satellite imagery that uses colorization as a pretext task. By training a model to predict visible RGB colors from non-visible spectral bands (like infrared), the authors create a feature extractor that understands the semantic relationship between land characteristics and their appearance. This method significantly boosts performance on downstream tasks like land-cover classification and disease detection, especially when labeled data is scarce.

What Problem Does This Paper Address?

Deep learning has revolutionized computer vision, but it is notoriously data-hungry. In Remote Sensing (RS), we have a massive influx of raw satellite data, but high-quality labels (e.g., "this pixel is a coniferous forest") are expensive and rare, requiring expert analysis or in-field measurements.

To get around this, researchers usually take models pre-trained on ImageNet (a dataset of everyday objects like cats and cars) and fine-tune them on satellite images. However, satellite imagery is fundamentally different: it’s captured from a top-down perspective and contains many more "colors" (spectral bands) than the standard Red, Green, and Blue. This domain gap means ImageNet features aren't always a perfect fit for Earth Observation.

Methods

The authors propose a two-stage pipeline centered on a novel self-supervised task:

  1. Self-Supervised Colorization: They use an encoder-decoder architecture (based on ResNet-18) where the input is the set of non-visible spectral bands (9 bands from Sentinel-2) and the target is the visible color information. Specifically, they predict the a and b chromaticity channels of the CIE Lab color space. The intuition is that to predict that a region should be "green," the network must learn to identify "vegetation" from the infrared bands.
  2. Fine-tuning: The trained encoder is then used as a starting point for specific tasks, such as classifying land types or predicting virus outbreaks.
  3. Ensemble Modeling: The authors discovered that features learned from RGB images (via ImageNet) and features learned from spectral bands (via colorization) are complementary. They built an ensemble that averages predictions from two branches—one fed with RGB and one with spectral bands—to get the best of both worlds.
Fig. 1. An overview of the proposed pipeline for feature learning on satellite imagery.

Datasets

  • BigEarthNet: A massive benchmark for land-cover classification containing over 500,000 Sentinel-2 image tiles. The authors used a refined version with 19 classes.
  • West Nile Disease (WND) Dataset: A specialized dataset pairing Sentinel-2 imagery with ground-truth virus reports across Italy. This represents a challenging, real-world "small data" scenario with only about 1,500 distinct cases.

Key Insights and Ideas

  • Semantics in the Spectrum: There is a strong link between spectral signatures and visible colors (e.g., water is blue, crops are green). Colorization forces the model to learn these semantic relationships without needing human-provided labels.
  • Diversity is Strength: Using Grad-CAM visualizations, the authors showed that the RGB branch and the Spectral branch focus on different parts of the input images. This "weak correlation" between their internal representations is exactly why the ensemble model works so well.
  • Efficiency at the Edge: The colorization-based features are particularly powerful when you only have a few thousand labeled examples, outperforming standard ImageNet pre-training in data-constrained environments.
Fig. 2. Some examples of the BigEarthNet dataset, coupled with the predicted colorization and visual explanations provided by the ensemble method for RGB and spectral inputs.

Key Results

  • Land-Cover Classification: On BigEarthNet, the colorization pre-training outperformed training from scratch and ImageNet pre-training when using small subsets of data (1k to 50k samples).
  • State-of-the-Art Performance: The proposed ensemble (ResNet-18 based) achieved an F1 score of 81.10% on BigEarthNet, outperforming much larger and more complex models like ResNet-152 (76.5%).
  • Disease Detection: For West Nile Virus detection, the ensemble model reached an F1 score of 0.852, a significant jump over single-modality baselines (see Table III).
Table showing performance gains on West Nile Disease detection

Limitations and Future Work

The authors noted a curious finding: when the entire BigEarthNet dataset (519k examples) is available for training, the advantage of colorization pre-training diminishes compared to standard ImageNet initialization on the spectral bands. They suggest this "saturation" effect warrants further investigation.

Additionally, they used a simple L1 loss for colorization to keep computation manageable for high-dimensional satellite data. Future work could explore more complex loss functions, like multinomial classification, which have shown promise in standard computer vision colorization tasks but are currently too computationally expensive for large-scale remote sensing.

Summary updated

Abstract

The recent growth in the number of satellite images fosters the development of effective deep-learning techniques for Remote Sensing (RS). However, their full potential is untapped due to the lack of large annotated datasets. Such a problem is usually countered by fine-tuning a feature extractor that is previously trained on the ImageNet dataset. Unfortunately, the domain of natural images differs from the RS one, which hinders the final performance. In this work, we propose to learn meaningful representations from satellite imagery, leveraging its high-dimensionality spectral bands to reconstruct the visible colors. We conduct experiments on land cover classification (BigEarthNet) and West Nile Virus detection, showing that colorization is a solid pretext task for training a feature extractor. Furthermore, we qualitatively observe that guesses based on natural images and colorization rely on different parts of the input. This paves the way to an ensemble model that eventually outperforms both the above-mentioned techniques.
BibTeX
@inproceedings{vincenzi2020color,
  author = {Stefano Vincenzi and Angelo Porrello and Pietro Buzzega and Marco Cipriano and Pietro Fronte and Roberto Cuccu and Carla Ippoliti and Annamaria Conte and Simone Calderara},
  title = {{The color out of space: learning self-supervised representations for Earth Observation imagery}},
  booktitle = {ICPR},
  year = {2020},
  doi = {10.1109/icpr48806.2021.9413112},
}
Figure