Crop Classification Under Varying Cloud Cover With Neural Ordinary Differential Equations

Nando Metzger, Mehmet Ozgur Turkoglu, Stefano D'Aronco, Jan Dirk Wegner, Konrad Schindler

TL;DR

This paper introduces a hybrid deep learning architecture that combines Neural Ordinary Differential Equations (NODEs) with Recurrent Neural Networks (RNNs) to classify crops from satellite imagery. By modeling the continuous growth cycle of plants, the "ODE-RNN" approach effectively handles the irregular time gaps caused by cloud cover, significantly outperforming standard models in sparse-data and early-season scenarios.

What Problem Does This Paper Address?

Optical satellites (like Sentinel-2) are essential for monitoring global agriculture, but they have a major weakness: they cannot see through clouds. While a satellite might pass over a field every five days, persistent cloud cover means the actual usable data is irregularly sampled in time.

Standard sequence models like LSTMs, GRUs, and Transformers are designed for discrete data sampled at regular intervals. They lack an internal "clock"; they know the order of observations but don't inherently understand the duration of the gaps between them. In agriculture, where crop reflectance changes drastically based on a continuous phenological (growth) cycle, ignoring the actual time elapsed leads to sub-optimal classification, especially when data is sparse.

Methods

The authors propose an ODE-RNN model that alternates between two distinct phases:

  1. The Prediction Step (NODE): Instead of a discrete jump between states, the model uses a Neural ODE to evolve the hidden state $h$ continuously over time. This represents the latent dynamics of the crop's growth. Because it is continuous, the model can "predict" the state of the system at any arbitrary point in time.
  2. The Update Step (RNN): When a new, cloud-free satellite observation becomes available, a standard recurrent unit (like a GRU or LSTM) "assimilates" this new data to update the hidden state.

Key Innovation: Unlike standard RNNs that simply ingest the next item in a list, the ODE-RNN uses the time difference ($\Delta t$) to determine how much the latent state should evolve before the next observation is processed.

Datasets

The models were evaluated on two large-scale multi-spectral Sentinel-2 time-series datasets:

  • TU Munich (TUM) Crop Data: Contains ~400,000 time series from North Munich, Germany, covering 19 crop classes. On average, 59% of the pixels are invalid due to clouds or missing data.
  • ZueriCrop Data: Contains ~350,000 time series from Zurich and Thurgau, Switzerland, covering 13 classes. Approximately 49% of the pixels are invalid.

Key Insights and Ideas

  • Natural Modeling: Modeling crop growth as a continuous process via differential equations is conceptually more aligned with biology than treating satellite snapshots as a simple discrete sequence.
  • The "Aha" of Sparsity: The model's advantage grows as the data gets worse. When observations are frequent, standard RNNs do fine. However, when cloud cover is heavy or training data is scarce, the "prior" knowledge of continuous dynamics provided by the ODE becomes a powerful regularizer.
  • Early-Season Forecasting: Because the NODE can extrapolate the hidden state into the future, the model is significantly better at "early classification"—predicting the crop type before the full growing season has concluded.
  • Parameter Efficiency: Interestingly, the ODE-RNN models achieved better results while using fewer parameters and less GPU memory than their standard RNN counterparts.

Key Results

  • Superior Accuracy: On the TUM dataset, the ODE-GRU achieved an 80.4% F1-score, outperforming standard GRUs (78.2%), TCNs (70.9%), and Transformers (79.8%).
  • Resilience to Clouds: In a "sparse" experiment where 75% of observations were removed, the ODE-RNN outperformed the baseline by 5.9% in F1-score.
  • Small Data Advantage: When trained on only 1% of the available training labels, the ODE-RNN maintained a 3.1% F1-score lead over the baseline, proving it is much more robust when ground-truth data is expensive to collect.
  • Early Classification: When only the first half of the season was available, the ODE-RNN provided a massive 13.5% boost in overall accuracy compared to a standard GRU.

Limitations and Future Work

  • Discontinuous Jumps: The authors note that the RNN update step causes a "jump" in the hidden state. This contradicts the ODE’s preference for smooth, continuous evolution. They suggest that future work could use Controlled Differential Equations (CDEs) to create a smoother integration of new data.
  • Update Mechanism: The current model uses a straightforward update; the authors suspect that a tighter integration between the prediction and update mechanisms (perhaps via Bayesian updates) could further improve performance.
  • Broader Application: While focused on crops, the authors believe this framework is a "straight-forward" win for almost any remote sensing task plagued by irregular data gaps, such as water quality monitoring or urban change detection.

Summary updated

Abstract

Optical satellite sensors cannot see the earth’s surface through clouds. Despite the periodic revisit cycle, image sequences acquired by earth observation satellites are, therefore, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">irregularly</i> sampled in time. State-of-the-art methods for crop classification (and other time-series analysis tasks) rely on techniques that implicitly assume regular temporal spacing between observations, such as recurrent neural networks (RNNs). We propose to use neural ordinary differential equations (NODEs) in combination with RNNs to classify crop types in irregularly spaced image sequences. The resulting ODE-RNN models consist of two steps: an update step, where a recurrent unit assimilates new input data into the model’s hidden state, and a prediction step, in which NODE propagates the hidden state until the next observation arrives. The prediction step is based on a continuous representation of the latent dynamics, which has several advantages. At the conceptual level, it is a more natural way to describe the mechanisms that govern the phenological cycle. From a practical point of view, it makes it possible to sample the system state at arbitrary points in time such that one can integrate observations whenever they are available and extrapolate beyond the last observation. Our experiments show that ODE-RNN, indeed, improves classification accuracy over common baselines, such as LSTM, GRU, temporal convolutional network, and transformer. The gains are most prominent in the challenging scenario where only few observations are available (i.e., frequent cloud cover). Moreover, we show that the ability to extrapolate translates to better classification performance early in the season, which is important for forecasting.
BibTeX
@article{metzger2022crop,
  author = {Nando Metzger and Mehmet Ozgur Turkoglu and Stefano D'Aronco and Jan Dirk Wegner and Konrad Schindler},
  title = {{Crop Classification Under Varying Cloud Cover With Neural Ordinary Differential Equations}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2022},
  doi = {10.1109/tgrs.2021.3101965},
}
Figure