Abstract
Satellite Earth observations (EO) can provide affordable and timely information for assessing crop conditions and food production. Such monitoring systems are essential in Africa, where there is high food insecurity and sparse agricultural statistics. EO-based monitoring systems require accurate cropland maps to provide information about croplands, but there is a lack of data to determine which of the many available land cover maps most accurately identify cropland in African countries. This study provides a quantitative evaluation and intercomparison of 11 publicly available land cover maps to assess their suitability for cropland classification and EO-based agriculture monitoring in Africa using statistically rigorous reference datasets from 8 countries. We hope the results of this study will help users determine the most suitable map for their needs and encourage future work to focus on resolving inconsistencies between maps and improving accuracy in low-accuracy regions.
Introduction
Sub-Saharan Africa (SSA) faces significant food security challenges, yet it lacks the robust agricultural statistics common in wealthier regions. Satellite Earth Observation (EO) offers a scalable solution, but monitoring crop health requires an accurate "cropland mask" to identify where crops are grown. While numerous global and regional land cover maps exist, they often disagree significantly. For policy-makers and researchers, choosing an inaccurate map can lead to skewed estimates of food production. This study addresses this "map paralysis" by providing a statistically rigorous, multi-country quantitative evaluation of 11 publicly available land cover products.
Methodology
The researchers conducted a quantitative intercomparison of 11 maps with resolutions ranging from 10 meters to 1,000 meters. All maps were resampled to a common 10m resolution and binarized into "crop" vs. "non-crop" based on each product's specific definitions. To evaluate performance, the study utilized a newly released reference dataset of 3,386 high-quality samples across Kenya, Malawi, Mali, Rwanda, Tanzania, Togo, Uganda, and Zambia. These samples were manually labeled by experts using multi-temporal, high-resolution imagery. Metrics included Accuracy, F1 scores, Precision (User’s Accuracy), and Recall (Producer’s Accuracy) to account for cropland often being a "rare class."
Key Insights: The Consensus Crisis
A primary finding of the study is the "consensus crisis." The 11 maps evaluated unanimously agree on cropland locations for less than 0.5% of the land in the study areas. Disagreement is most pronounced at field boundaries where land cover transitions are fuzzy.
Furthermore, the study debunked the "resolution myth," finding only a weak correlation between spatial resolution and performance. High-resolution (10m) maps are not inherently superior to 30m maps if the underlying models are not tuned for the heterogeneous landscapes of SSA. Regional models, such as Digital Earth Africa (DEA), often outperformed massive global models, suggesting that "one-size-fits-all" approaches struggle with regional agricultural nuances.
Performance Results
Performance varied significantly by geography. While maps performed relatively well in Rwanda (average F1 ~0.71), they struggled in Mali (average F1 ~0.21), where distinguishing sparse crops from natural semi-arid vegetation is extremely difficult.
Digital Earth Africa (DEA) and ESA WorldCover emerged as the most consistent top performers. DEA frequently achieved the highest scores across multiple metrics, while WorldCover led in mean precision. Despite these leaders, for 7 out of 8 countries, the average F1 score across all maps remained below 0.7, highlighting a persistent accuracy gap.
Conclusion and Future Directions
The small, fragmented nature of African fields remains a major challenge for satellite imagery, as many fields are smaller than a single 10m or 30m pixel. Future work should focus on targeted labeling in "zones of confusion," improving regional modeling to incorporate local phenology, and standardizing the definition of "cropland" across map producers to reduce semantic disagreement.
Summary updated