Efficient Representation Learning of Satellite Image Time Series and Their Fusion for Spatiotemporal Applications

Poonam Goyal, Arshveer Kaur, Arvind Ram, Navneet GoyalAAAI 2024 · DOI

TL;DR

This paper introduces PatchNet and FuSITSNet, novel deep learning models designed to efficiently process and fuse satellite image time series (SITS) from different sensors. By intelligently selecting informative image patches and learning joint representations, these models achieve state-of-the-art results in spatiotemporal applications like crop yield prediction, snow cover monitoring, and solar energy forecasting, while significantly reducing computational burden.

What Problem Does This Paper Address?

Monitoring Earth's surface using satellite imagery is crucial for various applications, but it's hampered by a fundamental trade-off: high spatial resolution (detailed images) often comes with low temporal resolution (infrequent revisits), and vice-versa. For instance, Landsat satellites offer high spatial detail but revisit every 16 days, while MODIS provides daily coverage but with much coarser resolution. This disparity makes it challenging to obtain SITS with both high spatial and temporal granularity, which is essential for accurately tracking dynamic phenomena. Processing the massive datasets generated by high-resolution satellites also presents a significant computational bottleneck, hindering the widespread adoption of these technologies. This paper aims to overcome these limitations by developing efficient methods for processing and fusing SITS from different sources.

Methods

The paper proposes two core models:

  • PatchNet: This model addresses the computational challenge of processing high-spatial-resolution SITS. Instead of processing entire images, PatchNet uses a "patch selection" mechanism. It intelligently selects a fraction (1/p) of image patches that are most informative for the task at hand. This selection is guided by a self-attention mechanism and a "neighbor selector" that ensures focus on relevant areas. PatchNet employs a Time Series Encoder (TSE) which includes a 3D CNN and a Spatial Attention Mask to extract features from these selected patches. This approach drastically reduces the amount of data that needs to be processed, leading to significant computational savings while maintaining high accuracy.
  • FuSITSNet: This model tackles the problem of fusing SITS from satellites with different spatial and temporal resolutions. It takes the learned representations from two encoders (PatchNet for high-spatial-resolution data like Landsat, and TSE for coarser-resolution data like MODIS) and fuses them using a "twofold fusion module." This module comprises:
  • Patch Alignment Module (PAM): This module aligns patches from the high-spatial-resolution SITS with corresponding regions in the coarser-resolution SITS, learning fine temporal patterns from the aligned data.
  • Cross-Modal Attention (CMA): This mechanism learns inter-modality relationships, allowing one modality to compensate for the weaknesses of the other and capturing complementary aspects. FuSITSNet learns a joint representation without generating intermediate images, thus avoiding data volume inflation and computational overhead associated with generative fusion methods.

Datasets

The models were trained and evaluated on datasets for three spatiotemporal applications:

  • Crop Yield Prediction (CYP): Data from the top corn and soybean producing counties in the United States. Crop yield labels were collected from USDA Quick Stats.
  • Snow Cover Prediction (SCP): Data from counties experiencing significant snowfall. Snow cover percentage was obtained from the MODIS MOD10A1 product.
  • Solar Energy Prediction (SEP): Data from five states.

The paper mentions using unsigned integers (uint) for data compression, reducing storage by four times, and that the difference in RMSE for CYP was less than 1% when comparing float vs. uint SITS. They used five common surface reflectance bands from MODIS and Landsat-8.

Key Insights and Ideas

  • Intelligent Patch Selection is Key: The paper demonstrates that not all parts of a satellite image time series are equally important for a given task. By intelligently selecting a subset of informative patches (PatchNet), significant computational savings can be achieved without sacrificing accuracy.
  • Fusion Without Generation: FuSITSNet's ability to fuse SITS from different resolutions at the feature level, without generating intermediate high-resolution images, is a major breakthrough. This avoids the computational cost and potential noise propagation associated with generative fusion models.
  • Complementary Strengths: The fusion module effectively leverages the strengths of different satellite sensors – the high spatial resolution of Landsat and the high temporal resolution of MODIS – to create a more comprehensive representation.
  • Democratizing Satellite Data: By reducing computational requirements and enabling effective fusion, these models make advanced spatiotemporal analysis more accessible to a wider range of researchers and applications.

Key Results

  • PatchNet Performance: PatchNet achieved state-of-the-art results for crop yield prediction compared to histogram-based models, outperforming them even without additional meteorological or soil data. When meteorological data was incorporated, PatchNet further improved performance.
  • FuSITSNet Superiority: FuSITSNet consistently outperformed single-modality baselines (Landsat-8 and MODIS individually) across all three applications (CYP, SCP, SEP). For example, in SEP, FuSITSNet achieved a significant improvement of approximately 76% compared to PatchNet (Landsat-8).
  • Generative Fusion Outperformed: FuSITSNet also outperformed existing generative fusion models (STARFM, RSFN, GAN) on enhanced Landsat-8 SITS, showing substantial RMSE reductions.
  • Computational Efficiency: FuSITSNet's running time is approximately 1/4th of other fusion models because it bypasses the generation step.
  • Optimal Patch Traversal: An ablation study indicated that traversing 1/4th of the SITS (p=4) with a patch size of 64x64 provided a good balance between computation time and RMSE for PatchNet.

Limitations and Future Work

  • Permutation Invariant Applications: The proposed approach is currently applied only to permutation-invariant applications (where pixel location doesn't matter for the final prediction).
  • Future Work: The authors plan to extend their approach to other applications, such as land use and land cover classification problems. They also suggest that incorporating meteorological data further improves the performance of FuSITSNet.

Summary updated

BibTeX
@inproceedings{goyal2024efficient,
  author = {Poonam Goyal and Arshveer Kaur and Arvind Ram and Navneet Goyal},
  title = {{Efficient Representation Learning of Satellite Image Time Series and Their Fusion for Spatiotemporal Applications}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i8.28686},
}
Figure