Towards Transformer-based Homogenization of Satellite Imagery for Landsat-8 and Sentinel-2

Venkatesh Thirugnana Sambandham, Konstantin Kirchheim, Sayan Mukhopadhaya, Frank Ortmeier

Abstract

Landsat-8 (NASA) and Sentinel-2 (ESA) are two prominent multi-spectral imaging satellite projects that provide publicly available data. The multi-spectral imaging sensors of the satellites capture images of the earth's surface in the visible and infrared region of the electromagnetic spectrum. Since the majority of the earth's surface is constantly covered with clouds, which are not transparent at these wavelengths, many images do not provide much information. To increase the temporal availability of cloud-free images of a certain area, one can combine the observations from multiple sources. However, the sensors of satellites might differ in their properties, making the images incompatible. This work provides a first glance at the possibility of using a transformer-based model to reduce the spectral and spatial differences between observations from both satellite projects. We compare the results to a model based on a fully convolutional UNet architecture. Somewhat surprisingly, we find that, while deep models outperform classical approaches, the UNet significantly outperforms the transformer in our experiments.

Introduction: The Challenge of Satellite Homogenization

To monitor Earth effectively—for instance, tracking crop health or deforestation—high-frequency data is essential. While individual satellite missions like NASA’s Landsat-8 and the ESA’s Sentinel-2 provide excellent data, their individual revisit times can be long. By combining their data, researchers could theoretically obtain updates every 2.3 days.

However, these datasets cannot be mixed directly. They possess different spatial resolutions (Sentinel-2 is sharper) and varying spectral sensitivities. This paper investigates homogenization: using deep learning to transform lower-resolution Landsat-8 images to match the high-resolution spectral characteristics of Sentinel-2.

Visual comparison of (a) Sentinel-2, (b) Landsat-8, and (c) Panchromatic imagery demonstrating the differences in resolution and spectral properties.

Methodology and Data

The study compares three distinct approaches to upscale and harmonize imagery:

  1. Bicubic Interpolation: A standard mathematical baseline for image resizing.
  2. UNet: A classic fully convolutional encoder-decoder architecture widely used in remote sensing.
  3. ESRT (Efficient Super Resolution Transformer): A modern architecture utilizing attention mechanisms to capture long-range dependencies. The authors modified ESRT by removing its internal upsampling block to accommodate pre-processed input images.

Dataset

The models were trained on a custom dataset curated from a large region across Germany, consisting of co-located Landsat-8 and Sentinel-2 image pairs taken on the same day. The dataset includes 63,740 training patches, each containing 7 channels: visible light (RGB), Near-Infrared (NIR), two Short-Wave Infrared (SWIR) bands, and the Landsat-8 panchromatic band.

Key Results

Performance was evaluated using the Structural Similarity Index Measure (SSIM) and Normalized Root Mean Squared Error (NRMSE).

Method SSIM (Higher is better) NRMSE (Lower is better)
Bicubic (Baseline) 0.8763 ± 0.043 0.1839 ± 0.056
ESRT (Transformer) 0.8898 ± 0.039 0.1705 ± 0.064
UNet (CNN) 0.9114 ± 0.028 0.1460 ± 0.048
Box plots showing the distribution of (a) NRMSE and (b) SSIM across the test set for the three evaluated methods.

While both deep learning models significantly improved spatial quality over the bicubic baseline, the UNet consistently outperformed the Transformer-based ESRT in both metrics.

Discussion: Why UNet Outperformed Transformers

The most significant finding was the superiority of the UNet over the Transformer. The authors suggest that the inductive bias of CNNs—specifically translation invariance—is more valuable for this specific task than the non-local interactions captured by Transformers. While Transformers excel in many computer vision benchmarks, the local receptive field of a deep CNN appears more effective for image-to-image translation in remote sensing.

Furthermore, the ESRT model exhibited spatial artifacts, particularly in the corners of image patches, which hindered its overall reliability compared to the smoother results produced by the UNet.

Conclusion and Future Work

Despite the success of deep learning models, all approaches struggled with high-frequency components, such as complex urban structures and human settlements. Future research will explore hybrid architectures that combine the strengths of both CNNs and Transformers, while also focusing on reducing the artifacts observed in Transformer-based outputs.

Summary updated

Abstract

Landsat-8 (NASA) and Sentinel-2 (ESA) are two prominent multi-spectral imaging satellite projects that provide publicly available data. The multi-spectral imaging sensors of the satellites capture images of the earth's surface in the visible and infrared region of the electromagnetic spectrum. Since the majority of the earth's surface is constantly covered with clouds, which are not transparent at these wavelengths, many images do not provide much information. To increase the temporal availability of cloud-free images of a certain area, one can combine the observations from multiple sources. However, the sensors of satellites might differ in their properties, making the images incompatible. This work provides a first glance at the possibility of using a transformer-based model to reduce the spectral and spatial differences between observations from both satellite projects. We compare the results to a model based on a fully convolutional UNet architecture. Somewhat surprisingly, we find that, while deep models outperform classical approaches, the UNet significantly outperforms the transformer in our experiments.
BibTeX
@misc{sambandham2022towards,
  author = {Venkatesh Thirugnana Sambandham and Konstantin Kirchheim and Sayan Mukhopadhaya and Frank Ortmeier},
  title = {{Towards Transformer-based Homogenization of Satellite Imagery for Landsat-8 and Sentinel-2}},
  year = {2022},
  eprint = {2210.07654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.07654},
}

Loading related content...

Figure