End-to-End Learned Early Classification of Time Series for In-Season Crop Type Mapping

Marc Rußwurm, Nicolas Courty, Rémi Emonet, Sébastien Lefèvre, Devis Tuia, Romain Tavenard

TL;DR

ELECTS is a modular deep learning framework that enables satellite-based crop classification models to make accurate predictions much earlier in the growing season. By adding a "stopping head" and a specialized loss function, the system learns to halt data processing the moment it reaches high confidence, matching state-of-the-art accuracy while using as little as 16% of the full-season data.

What Problem Does This Paper Address?

In agricultural monitoring, timing is everything. Governments need to verify crop subsidies, and aid organizations need to predict food shortages months before harvest. Traditionally, remote sensing experts have relied on "full-season" classification, where they wait for a complete year of satellite imagery to produce a map.

While "in-season" mapping exists, it usually involves "incremental classification"—re-fitting models every time a new image arrives—which is computationally expensive and often relies on hand-crafted heuristics. The challenge is finding the "sweet spot": how do we train a model to be as accurate as possible, as early as possible, without manually telling it which biological features (phenology) to look for?

Methods

The authors introduce ELECTS (End-to-End Learned Early Classification of Time Series). The beauty of ELECTS is its modularity; it isn't a standalone model but a framework that can be "plugged into" almost any deep time-series classifier (like LSTMs, Transformers, or Temporal CNNs).

Key components include:

  • Two-Headed Architecture: The model features a standard classification head (predicting the crop type) and a new stopping head. The stopping head outputs a probability ($d_t$) that the model has seen enough data to make a final call.
  • Joint Loss Function: The model is trained using a loss function that balances two competing objectives: accuracy (don't be wrong) and an "earliness reward" (don't be slow).
  • The $\epsilon$ Offset: A technical highlight is the inclusion of a small constant offset ($\epsilon$) in the stopping probability calculation. This prevents the model from falling into a "lazy" local minimum where it simply stops at the very first time step to maximize the earliness reward at the cost of all accuracy.

Datasets

The researchers evaluated ELECTS across four diverse datasets to prove its global applicability:

  1. BreizhCrops (France): A massive dataset of over 600,000 field parcels using Sentinel-2 imagery.
  2. BavarianCrops (Germany): A large-scale dataset covering 7 common crop types in Southern Germany.
  3. Ghana & South Sudan: Two smaller-scale datasets (from the SustainBench archive) that combine Sentinel-1 (radar), Sentinel-2 (optical), and PlanetScope data. These represent the challenge of mapping in regions with smaller fields and less labeled training data.

Key Insights and Ideas

  • Learning Biology Automatically: Perhaps the most striking finding is that ELECTS "discovered" plant phenology on its own. For example, the model learned to stop and classify rapeseed fields in late April—exactly when their characteristic yellow blossoms appear. It learned to classify barley only after harvest when the bare soil became visible. It did this without any human-provided crop calendars.
  • Accuracy vs. Confidence: The model demonstrates that early-stopped fields are actually more accurate than the average. By allowing the model to choose its own stopping point, it naturally prioritizes "easy" fields early on and waits for more evidence on ambiguous ones.
  • Massive Resource Savings: The authors calculate that if scaled to the 43TB of Sentinel-2 data acquired over Europe annually, ELECTS could save 26TB of downloading and processing by stopping early.

Key Results

  • Efficiency: ELECTS achieved state-of-the-art accuracy while using only 32% to 40% of the time series in Europe, and as little as 16% to 20% in Africa.
  • Performance:
    • In Germany, it reached 86% accuracy (matching full-season models) with an average stopping date in mid-May.
  • In South Sudan, it reached 83% accuracy by early March (day of year 61).
  • Stability: Unlike previous early-classification methods, ELECTS proved stable across different regions and dataset sizes without needing site-specific hyperparameter tuning.

Limitations and Future Work

  • Label Imbalance: Like most deep learning models, ELECTS struggles with rare crop classes. If a class is infrequent, the model is less penalized for getting it wrong, which can lead to poor performance on niche crops.
  • Data Hunger: While it performed well in Africa, the authors note that deep learning inherently requires large amounts of annotated data, which can be a barrier in developing regions.
  • Future Directions: The authors suggest extending ELECTS to spatio-temporal data (looking at neighboring pixels simultaneously) by swapping the LSTM feature extractor for a 2D-convolutional architecture.

Source Paper: Rußwurm, M., et al. (2022). End-to-End Learned Early Classification of Time Series for In-Season Crop Type Mapping. ISPRS Journal of Photogrammetry and Remote Sensing. Code: https://github.com/marccoru/elects

Summary updated

Abstract

Remote sensing satellites capture the cyclic dynamics of our Planet in regular time intervals recorded in satellite time series data. End-to-end trained deep learning models use this time series data to make predictions at a large scale, for instance, to produce up-to-date crop cover maps. Most time series classification approaches focus on the accuracy of predictions. However, the earliness of the prediction is also of great importance since coming to an early decision can make a crucial difference in time-sensitive applications. In this work, we present an End-to-End Learned Early Classification of Time Series (ELECTS) model that estimates a classification score and a probability of whether sufficient data has been observed to come to an early and still accurate decision. ELECTS is modular: any deep time series classification model can adopt the ELECTS conceptual idea by adding a second prediction head that outputs a probability of stopping the classification. The ELECTS loss function then optimizes the overall model on a balanced objective of earliness and accuracy. Our experiments on four crop classification datasets from Europe and Africa show that ELECTS allows reaching state-of-the-art accuracy while reducing the quantity of data massively to be downloaded, stored, and processed. The source code is available at https://github.com/marccoru/elects.
BibTeX
@inproceedings{ruwurm2023end,
  author = {Marc Rußwurm and Nicolas Courty and Rémi Emonet and Sébastien Lefèvre and Devis Tuia and Romain Tavenard},
  title = {{End-to-End Learned Early Classification of Time Series for In-Season Crop Type Mapping}},
  booktitle = {ISPRS},
  year = {2023},
  doi = {10.1016/j.isprsjprs.2022.12.016},
}
Figure