How accurate are existing land cover maps for agriculture in Sub-Saharan Africa?

Hannah Kerner, Catherine Nakalembe, Adam Yang, Ivan Zvonkov, Ryan McWeeny, Gabriel Tseng, Inbal Becker-Reshef

Abstract

Satellite Earth observations (EO) can provide affordable and timely information for assessing crop conditions and food production. Such monitoring systems are essential in Africa, where there is high food insecurity and sparse agricultural statistics. EO-based monitoring systems require accurate cropland maps to provide information about croplands, but there is a lack of data to determine which of the many available land cover maps most accurately identify cropland in African countries. This study provides a quantitative evaluation and intercomparison of 11 publicly available land cover maps to assess their suitability for cropland classification and EO-based agriculture monitoring in Africa using statistically rigorous reference datasets from 8 countries. We hope the results of this study will help users determine the most suitable map for their needs and encourage future work to focus on resolving inconsistencies between maps and improving accuracy in low-accuracy regions.

Introduction

Sub-Saharan Africa (SSA) faces significant food security challenges, yet it lacks the robust agricultural statistics common in wealthier regions. Satellite Earth Observation (EO) offers a scalable solution, but monitoring crop health requires an accurate "cropland mask" to identify where crops are grown. While numerous global and regional land cover maps exist, they often disagree significantly. For policy-makers and researchers, choosing an inaccurate map can lead to skewed estimates of food production. This study addresses this "map paralysis" by providing a statistically rigorous, multi-country quantitative evaluation of 11 publicly available land cover products.

Methodology

The researchers conducted a quantitative intercomparison of 11 maps with resolutions ranging from 10 meters to 1,000 meters. All maps were resampled to a common 10m resolution and binarized into "crop" vs. "non-crop" based on each product's specific definitions. To evaluate performance, the study utilized a newly released reference dataset of 3,386 high-quality samples across Kenya, Malawi, Mali, Rwanda, Tanzania, Togo, Uganda, and Zambia. These samples were manually labeled by experts using multi-temporal, high-resolution imagery. Metrics included Accuracy, F1 scores, Precision (User’s Accuracy), and Recall (Producer’s Accuracy) to account for cropland often being a "rare class."

Key Insights: The Consensus Crisis

A primary finding of the study is the "consensus crisis." The 11 maps evaluated unanimously agree on cropland locations for less than 0.5% of the land in the study areas. Disagreement is most pronounced at field boundaries where land cover transitions are fuzzy.

Table showing the percentage of pixels where maps agree or disagree on cropland classification across eight countries.

Furthermore, the study debunked the "resolution myth," finding only a weak correlation between spatial resolution and performance. High-resolution (10m) maps are not inherently superior to 30m maps if the underlying models are not tuned for the heterogeneous landscapes of SSA. Regional models, such as Digital Earth Africa (DEA), often outperformed massive global models, suggesting that "one-size-fits-all" approaches struggle with regional agricultural nuances.

Performance Results

Performance varied significantly by geography. While maps performed relatively well in Rwanda (average F1 ~0.71), they struggled in Mali (average F1 ~0.21), where distinguishing sparse crops from natural semi-arid vegetation is extremely difficult.

Quantitative evaluation of 11 land cover maps across eight countries using Accuracy, F1, Precision, and Recall metrics.

Digital Earth Africa (DEA) and ESA WorldCover emerged as the most consistent top performers. DEA frequently achieved the highest scores across multiple metrics, while WorldCover led in mean precision. Despite these leaders, for 7 out of 8 countries, the average F1 score across all maps remained below 0.7, highlighting a persistent accuracy gap.

Conclusion and Future Directions

The small, fragmented nature of African fields remains a major challenge for satellite imagery, as many fields are smaller than a single 10m or 30m pixel. Future work should focus on targeted labeling in "zones of confusion," improving regional modeling to incorporate local phenology, and standardizing the definition of "cropland" across map producers to reduce semantic disagreement.

Summary updated

Abstract

Satellite Earth observations (EO) can provide affordable and timely information for assessing crop conditions and food production. Such monitoring systems are essential in Africa, where there is high food insecurity and sparse agricultural statistics. EO-based monitoring systems require accurate cropland maps to provide information about croplands, but there is a lack of data to determine which of the many available land cover maps most accurately identify cropland in African countries. This study provides a quantitative evaluation and intercomparison of 11 publicly available land cover maps to assess their suitability for cropland classification and EO-based agriculture monitoring in Africa using statistically rigorous reference datasets from 8 countries. We hope the results of this study will help users determine the most suitable map for their needs and encourage future work to focus on resolving inconsistencies between maps and improving accuracy in low-accuracy regions.
BibTeX
@inproceedings{kerner2024how,
  author = {Hannah Kerner and Catherine Nakalembe and Adam Yang and Ivan Zvonkov and Ryan McWeeny and Gabriel Tseng and Inbal Becker-Reshef},
  title = {{How accurate are existing land cover maps for agriculture in Sub-Saharan Africa?}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41597-024-03306-z},
}
Figure