Fine-grained Hierarchical Crop Type Classification from Integrated Hyperspectral EnMAP Data and Multispectral Sentinel-2 Time Series: A Large-scale Dataset and Dual-stream Transformer Method

Wenyuan Li, Shunlin Liang, Yuxiang Zhang, Liqin Liu, Keyan Chen, Yongzhe Chen, Han Ma, Jianglei Xu, Yichuan Ma, Shikang Guan, Zhenwei ShiarXiv 2025 · DOI · arXiv

Introduction

Accurate crop mapping is a cornerstone of global food security and precision agriculture. However, remote sensing often faces a trade-off: multispectral satellites like Sentinel-2 offer high temporal frequency (capturing phenology) but limited spectral resolution, while hyperspectral missions like EnMAP provide rich chemical "fingerprints" but infrequent revisits. This paper introduces H2Crop, a large-scale dataset that integrates these two modalities, alongside a dual-stream Transformer architecture designed to exploit both temporal growth patterns and fine-grained spectral signatures.

The H2Crop Dataset

The authors present H2Crop, a benchmark dataset covering over 1.2 million annotated field parcels across France. Unlike previous datasets that focus on a single modality or small geographic areas, H2Crop combines 30m EnMAP hyperspectral imagery (218 bands) with 10m Sentinel-2 monthly composites.

Comparison of H2Crop with existing remote sensing datasets, highlighting its scale and multi-modal nature.

A key feature of H2Crop is its hierarchical labeling system, which follows a four-level taxonomic structure (ranging from 6 broad classes to 101 specific crop types). This structure allows models to learn logical agricultural chains, such as identifying a parcel as "Arable Land" before refining it to "Cereal" and finally "Winter Soft Wheat."

Spatial distribution of H2Crop across France, showing hierarchical labels and corresponding spectral curves.

Methodology: Dual-stream Transformer

To process this multi-resolution, multi-modal data, the researchers developed a specialized architecture:

  1. Spectral-Spatial Decoupled ViT: This branch handles EnMAP data using parallel Transformers to isolate biochemical spectral signatures from spatial field geometries.
  2. MSI Temporal Transformer: A Video Swin Transformer processes 12-month Sentinel-2 sequences to capture seasonal phenological changes.
  3. Hierarchical Classification Head: A cascaded head predicts labels across four levels, utilizing historical crop maps from the previous year as context to account for typical crop rotation patterns.
  4. Cross-Resolution Fusion: Features are aligned using "pixel shuffle" upsampling, ensuring the 10m spatial detail of Sentinel-2 is preserved alongside the 30m spectral richness of EnMAP.

Key Insights and Results

The integration of hyperspectral data proves most beneficial at finer taxonomic levels. While multispectral data is sufficient for broad land-cover classification, hyperspectral bands provided a 6.3% F1-score boost for distinguishing specific crop species (Level 3).

Overall, the dual-stream Transformer achieved a 4.2% improvement in average F1-score over Sentinel-2 baselines and outperformed state-of-the-art architectures like 3DCNNs and CNN-LSTMs by approximately 2.6 percentage points. Furthermore, the model demonstrated strong "in-season" performance, achieving high accuracy with only the first six months of data (January–June), which is critical for early yield forecasting.

Conclusion and Future Directions

H2Crop represents a significant step forward in large-scale agricultural monitoring. While the current fusion method relies on concatenation, future work could explore cross-modal attention to better capture the co-evolution of spectral signatures and phenology. Additionally, addressing the extreme class imbalance inherent in real-world agricultural data—where common staples like wheat vastly outnumber specialty crops—remains a priority for improving model robustness.

Summary updated

BibTeX
@misc{li2025fine,
  author = {Wenyuan Li and Shunlin Liang and Yuxiang Zhang and Liqin Liu and Keyan Chen and Yongzhe Chen and Han Ma and Jianglei Xu and Yichuan Ma and Shikang Guan and Zhenwei Shi},
  title = {{Fine-grained Hierarchical Crop Type Classification from Integrated Hyperspectral EnMAP Data and Multispectral Sentinel-2 Time Series: A Large-scale Dataset and Dual-stream Transformer Method}},
  year = {2025},
  eprint = {2506.06155},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.06155},
}
Figure