Learning Disentangled Representations of Satellite Image Time Series

Eduardo Hugo Sanchez, Mathieu Serrurier, Mathias Ortner

Abstract

Satellite imagery provides a continuous stream of data, yet the vast majority remains unlabeled due to the high cost of manual annotation. This paper introduces an unsupervised framework designed to disentangle satellite image time series into two distinct components: a shared representation capturing static geographical features and an exclusive representation capturing time-specific dynamic changes. By combining Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), the proposed model serves as a versatile feature extractor. Experimental results demonstrate that these disentangled representations excel in downstream tasks such as land-cover classification, image retrieval, and change detection, achieving performance competitive with supervised methods.

Introduction

Satellite missions like Sentinel-2 generate massive datasets that are inherently complex, blending static information (land cover, urban layout) with dynamic signals (seasonal changes, human activity). Traditional supervised learning approaches are bottlenecked by the need for expensive human labels. This research addresses the "disentanglement" problem: automatically separating these two signals in an unsupervised manner. Successfully isolating "shared" features allows for consistent location identification across time, while "exclusive" features enable precise pinpointing of temporal events like deforestation or harvesting.

Methods

The proposed architecture is based on cross-domain autoencoders, utilizing a framework inspired by multimodal image-to-image translation.

Architecture and the "Swap" Trick

The model comprises four primary components:

  1. Shared Encoder ($E_{sh}$): Captures features common across a time series (e.g., topography).
  2. Exclusive Encoder ($E_{ex}$): Captures features unique to a specific timestamp (e.g., vegetation color).
  3. Decoder/Generator ($G$): Reconstructs a scene by combining a shared feature from one image with an exclusive feature from another.
  4. Discriminator ($D$): Ensures the visual realism of generated images.

To enforce disentanglement, the model is trained on image pairs ($x, y$) from the same location at different times. The generator must produce image $y$ by merging the shared features of $x$ with the exclusive features of $y$. This "swap trick" prevents the shared encoder from encoding time-specific information, as doing so would lead to reconstruction failure when applied across different time steps.

Figure 1: The proposed unsupervised framework for disentangling shared and exclusive features through a cross-domain autoencoder architecture.

Loss Functions

The training objective involves a minimax game using:

  • $L_1$ Reconstruction Loss: For pixel-level accuracy.
  • Shared Feature Loss: To minimize the distance between $E_{sh}(x)$ and $E_{sh}(y)$.
  • KL Divergence: To regularize the exclusive representation.
  • GAN Loss: To ensure high-quality image synthesis.

Datasets and Evaluation

The model was primarily trained on a Sentinel-2 dataset consisting of 1,050 time series, each containing 12 images (RGBI bands at 10m resolution). For evaluation, the EuroSAT dataset, containing 27,000 labeled images across 10 land-cover classes, was used to test the shared representation's utility.

Figure 2: The data pipeline illustrating the random selection of time series and image pairs (x, y) for unsupervised training.

Key Results

The disentangled latent spaces proved highly effective for various remote sensing tasks:

  • Land-Cover Classification: Using the unsupervised shared encoder as a feature extractor for EuroSAT yielded a 92.38% accuracy with minimal fine-tuning, reaching 94.54% with full training.
  • Image Retrieval: Shared features achieved a Recall@1 of 0.7372 when matching patches across time, significantly outperforming raw pixel comparisons (0.5083).
  • Change Detection: Qualitative analysis showed that the $L_1$ distance in the exclusive latent space effectively highlights land-use changes while remaining invariant to seasonal noise.
  • Image Translation: The model successfully synthesized realistic images by swapping temporal features, such as applying a "harvested" state to a "growing" field's geometry.

Conclusion and Future Work

This work demonstrates that temporal consistency in satellite imagery is a powerful proxy for human labels. By separating geographical and temporal signals, a single unsupervised model can support diverse applications. However, the current generator lacks some fine spatial details due to the absence of skip connections—a design choice made to preserve the information bottleneck. Future research will explore more sophisticated change detection metrics and the application of this framework to sub-meter resolution imagery.

Summary updated

BibTeX
@inproceedings{sanchez2019learning,
  author = {Eduardo Hugo Sanchez and Mathieu Serrurier and Mathias Ortner},
  title = {{Learning Disentangled Representations of Satellite Image Time Series}},
  booktitle = {ECML/PKDD},
  year = {2019},
  doi = {10.1007/978-3-030-46133-1\_19},
}
Figure