Introduction
Accurate crop mapping is a cornerstone of global food security and precision agriculture. However, remote sensing often faces a trade-off: multispectral satellites like Sentinel-2 offer high temporal frequency (capturing phenology) but limited spectral resolution, while hyperspectral missions like EnMAP provide rich chemical "fingerprints" but infrequent revisits. This paper introduces H2Crop, a large-scale dataset that integrates these two modalities, alongside a dual-stream Transformer architecture designed to exploit both temporal growth patterns and fine-grained spectral signatures.
The H2Crop Dataset
The authors present H2Crop, a benchmark dataset covering over 1.2 million annotated field parcels across France. Unlike previous datasets that focus on a single modality or small geographic areas, H2Crop combines 30m EnMAP hyperspectral imagery (218 bands) with 10m Sentinel-2 monthly composites.
A key feature of H2Crop is its hierarchical labeling system, which follows a four-level taxonomic structure (ranging from 6 broad classes to 101 specific crop types). This structure allows models to learn logical agricultural chains, such as identifying a parcel as "Arable Land" before refining it to "Cereal" and finally "Winter Soft Wheat."
Methodology: Dual-stream Transformer
To process this multi-resolution, multi-modal data, the researchers developed a specialized architecture:
- Spectral-Spatial Decoupled ViT: This branch handles EnMAP data using parallel Transformers to isolate biochemical spectral signatures from spatial field geometries.
- MSI Temporal Transformer: A Video Swin Transformer processes 12-month Sentinel-2 sequences to capture seasonal phenological changes.
- Hierarchical Classification Head: A cascaded head predicts labels across four levels, utilizing historical crop maps from the previous year as context to account for typical crop rotation patterns.
- Cross-Resolution Fusion: Features are aligned using "pixel shuffle" upsampling, ensuring the 10m spatial detail of Sentinel-2 is preserved alongside the 30m spectral richness of EnMAP.
Key Insights and Results
The integration of hyperspectral data proves most beneficial at finer taxonomic levels. While multispectral data is sufficient for broad land-cover classification, hyperspectral bands provided a 6.3% F1-score boost for distinguishing specific crop species (Level 3).
Overall, the dual-stream Transformer achieved a 4.2% improvement in average F1-score over Sentinel-2 baselines and outperformed state-of-the-art architectures like 3DCNNs and CNN-LSTMs by approximately 2.6 percentage points. Furthermore, the model demonstrated strong "in-season" performance, achieving high accuracy with only the first six months of data (January–June), which is critical for early yield forecasting.
Conclusion and Future Directions
H2Crop represents a significant step forward in large-scale agricultural monitoring. While the current fusion method relies on concatenation, future work could explore cross-modal attention to better capture the co-evolution of spectral signatures and phenology. Additionally, addressing the extreme class imbalance inherent in real-world agricultural data—where common staples like wheat vastly outnumber specialty crops—remains a priority for improving model robustness.
Summary updated