Recurrent Generative Networks for Multi-Resolution Satellite Data: An Application in Cropland Monitoring

Xiaowei Jia, Mengdie Wang, Ankush Khandelwal, Anuj Karpatne, Vipin KumarIJCAI 2019 · OpenReview

Abstract

Effective and timely monitoring of croplands is critical for managing food supply. While remote sensing data from earth-observing satellites can be used to monitor croplands over large regions, this task is challenging for small-scale croplands as they cannot be captured precisely using coarse-resolution data. On the other hand, the remote sensing data in higher resolution are collected less frequently and contain missing or disturbed data. Hence, traditional sequential models cannot be directly applied on high-resolution data to extract temporal patterns, which are essential to identify crops. In this work, we propose a generative model to combine multi-scale remote sensing data to detect croplands at high resolution. During the learning process, we leverage the temporal patterns learned from coarse-resolution data to generate missing high-resolution data. Additionally, the proposed model can track classification confidence in real time and potentially lead to an early detection. The evaluation in an intensively cultivated region demonstrates the effectiveness of the proposed method in cropland detection.

Introduction

Global food security relies heavily on the ability to monitor croplands accurately and in a timely manner. Remote sensing via satellites provides a scalable solution, yet researchers face a persistent "resolution trade-off." Coarse-resolution data, such as MODIS (250m pixels), offers daily updates ideal for capturing rapid crop growth cycles but lacks the spatial detail to resolve small-scale fields. Conversely, high-resolution data like Sentinel-2 (10m pixels) can distinguish individual fields but suffers from low temporal frequency and frequent data gaps caused by cloud cover or atmospheric interference.

Traditional sequential models, such as standard Recurrent Neural Networks (RNNs), struggle with these irregular gaps and the inherent stochasticity of crop growth—where identical crops may exhibit different spectral signatures due to local environmental factors. To address these challenges, we introduce MARSM (Multi-scale Analysis of Remote Sensing data with Missing or poor quality data), a generative framework designed to fuse multi-resolution streams for robust cropland monitoring.

Methodology: The MARSM Framework

The MARSM framework leverages Variational Recurrent Neural Networks (VRNNs) to model the temporal dynamics of crop growth. Unlike deterministic LSTMs, VRNNs incorporate latent random variables to capture the complex variability of biological processes, preventing the "blurry" predictions often seen in standard regression models.

Multi-Scale Fusion and Data Generation

The core innovation of MARSM lies in its hierarchical structure. The model processes two parallel data streams:

  1. Coarse Guidance: It learns broad temporal "templates" from high-frequency MODIS data.
  2. Fine Reconstruction: These templates guide the generation of missing high-resolution Sentinel data. Even when a Sentinel image is unavailable, the corresponding MODIS data provides sufficient context to infer the field's state.
The MARSM architecture fuses MODIS and Sentinel data, using learned temporal patterns to impute missing high-resolution observations.

To ensure the generated data is physically plausible, an adversarial regularizer is employed. A discriminator network attempts to distinguish between real and imputed Sentinel sequences, forcing the generator to produce realistic transitions. Furthermore, the model performs spatial smoothing on the latent variables derived from MODIS, creating a more robust representation than raw pixel-level smoothing.

Experimental Setup

The model was evaluated in an intensively cultivated 4,900-hectare region in southwestern Minnesota, USA. The dataset included:

  • MODIS (Coarse): 250m resolution, 10-day composites.
  • Sentinel-2A (High-Res): 10m resolution across 13 spectral bands.
  • Ground Truth: USDA Crop Data Layer labels for Corn, Soybean, and Sugarbeet.

Results and Discussion

MARSM demonstrates significant improvements in both reconstruction and classification tasks. Quantitatively, the model achieved an overall F1-score of 0.96, outperforming standard LSTMs (0.88) and specialized models for missing data like GRU-D (0.92).

Robustness and Early Detection

A key strength of MARSM is its resilience to data gaps. During early spring, when cloud cover is frequent, MARSM maintains high performance while baseline models degrade. The adversarial component plays a crucial role here; as shown in the performance analysis, the full MARSM model maintains lower error distances across time steps compared to variants without adversarial training or spatial normalization.

Comparison of average absolute distance across time steps for MARSM and its variants, highlighting the benefit of the full generative framework.

The model also enables progressive classification. By accumulating discriminative information at each time step, it can track classification confidence in real-time. For instance, corn can often be identified earlier than soybeans because it exhibits a faster "green-up" phase in June—a biological reality the model captures mathematically.

Conclusion

MARSM effectively bridges the gap between high-frequency coarse data and low-frequency high-resolution data. By treating missing data as a generative task guided by multi-scale context, it provides a robust tool for early-season crop identification. Future work will focus on improving regional generalizability to account for varying soil types and farming practices, as well as optimizing the computational efficiency of the adversarial training process.

Summary updated

Abstract

Effective and timely monitoring of croplands is critical for managing food supply. While remote sensing data from earth-observing satellites can be used to monitor croplands over large regions, this task is challenging for small-scale croplands as they cannot be captured precisely using coarse-resolution data. On the other hand, the remote sensing data in higher resolution are collected less frequently and contain missing or disturbed data. Hence, traditional sequential models cannot be directly applied on high-resolution data to extract temporal patterns, which are essential to identify crops. In this work, we propose a generative model to combine multi-scale remote sensing data to detect croplands at high resolution. During the learning process, we leverage the temporal patterns learned from coarse-resolution data to generate missing high-resolution data. Additionally, the proposed model can track classification confidence in real time and potentially lead to an early detection. The evaluation in an intensively cultivated region demonstrates the effectiveness of the proposed method in cropland detection.
BibTeX
@inproceedings{jia2019recurrent,
  author = {Xiaowei Jia and Mengdie Wang and Ankush Khandelwal and Anuj Karpatne and Vipin Kumar},
  title = {{Recurrent Generative Networks for Multi-Resolution Satellite Data: An Application in Cropland Monitoring}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=U92r85lARRO},
}
Figure