How accurate are existing land cover maps for agriculture in Sub-Saharan Africa?

Hannah Kerner, Catherine Nakalembe, Adam Yang, Ivan Zvonkov, R. McWeeny, Gabriel Tseng, Inbal Becker‐Reshef

Abstract

Satellite Earth observations (EO) can provide affordable and timely information for assessing crop conditions and food production. Such monitoring systems are essential in Africa, where food insecurity is high and agricultural statistics are sparse. EO-based monitoring systems require accurate cropland maps to provide information about croplands, but there is a lack of data to determine which of the many available land cover maps most accurately identify cropland in African countries. This study provides a quantitative evaluation and intercomparison of 11 publicly available land cover maps to assess their suitability for cropland classification and EO-based agriculture monitoring in Africa using statistically rigorous reference datasets from 8 countries. We hope the results of this study will help users determine the most suitable map for their needs and encourage future work to focus on resolving inconsistencies between maps and improving accuracy in low-accuracy regions.

Introduction

Sub-Saharan Africa remains a global hotspot for food insecurity, yet it suffers from a chronic deficit of ground-level agricultural statistics. Satellite Earth Observation (EO) has emerged as a primary tool for filling this data gap. However, the effectiveness of EO-based monitoring depends entirely on the accuracy of "cropland masks"—maps that define where crops are actually grown. While numerous global and regional land cover maps exist, they often provide wildly different estimates for the same area, leaving researchers and policy-makers without a clear guide on which product to trust.

This study presents the first rigorous, multi-country intercomparison of 11 major satellite-based cropland maps for Sub-Saharan Africa. By introducing a high-quality reference dataset of over 3,300 samples, the researchers reveal a startling lack of consensus between existing products and provide a framework for navigating this "overwhelming array" of data.

Methodology and Datasets

The authors evaluated 11 publicly available land cover maps, ranging in resolution from 10m/pixel (e.g., Esri LULC, ESA WorldCover) to 1000m/pixel (ASAP Crop Mask). To assess these maps, a new benchmark dataset was developed consisting of 3,386 samples across eight countries: Kenya, Malawi, Mali, Rwanda, Tanzania, Togo, Uganda, and Zambia. These points were manually labeled by experts using high-resolution PlanetScope (3m) and Sentinel-2 (10m) imagery.

The evaluation framework utilized standard metrics: Accuracy, F1 score, Precision (user’s accuracy), and Recall (producer’s accuracy). Additionally, the study performed consensus analysis to visualize spatial agreement and investigated whether higher spatial resolution or temporal proximity to the reference year improved map performance.

Key Findings

The Consensus Crisis

A primary finding of the study is the massive disparity in how different models define and detect African smallholder agriculture. Unanimous agreement on cropland was found in fewer than 0.5% of pixels in every country studied. Disagreement is particularly prevalent at field boundaries and in "mosaic" landscapes where crops are intermingled with natural vegetation.

Spatial agreement map showing the number of products predicting cropland. Red indicates unanimous agreement on non-crop; yellow/green indicates high disagreement.

Performance Variance

Performance varied drastically by geography and metric. For instance, the average F1 score in Mali was a dismal 0.21, compared to 0.71 in Rwanda. This suggests that global models still struggle with the diverse agro-ecological landscapes of Africa. Interestingly, higher spatial resolution was only weakly correlated with better performance; while it helped reduce false positives (improving precision), it did not necessarily improve the detection of actual cropland (recall).

Comparison of performance metrics (F1, Accuracy, Recall, Precision) across 11 land cover maps and 8 African countries.

Top Performers

Regional models trained specifically for the African continent generally outperformed one-size-fits-all global models. Digital Earth Africa and ESA WorldCover most frequently achieved the highest scores across metrics and countries. While a "Majority Vote" ensemble performed better than most individual maps, it was still often surpassed by the best regional products.

Conclusion and Future Directions

The study underscores that no single map is "best" for every metric or country. The authors advocate for targeted improvements in low-performing regions like Mali and a greater focus on resolving field boundaries. Future work should prioritize expanding this evaluation to the entire continent and incorporating temporal features—tracking land changes over a season—to better distinguish crops from other vegetation types. This research provides a vital roadmap for users to select the most suitable cropland maps for food security monitoring in Africa.

Summary updated

Abstract

Satellite Earth observations (EO) can provide affordable and timely information for assessing crop conditions and food production. Such monitoring systems are essential in Africa, where food insecurity is high and agricultural statistics are sparse. EO-based monitoring systems require accurate cropland maps to provide information about croplands, but there is a lack of data to determine which of the many available land cover maps most accurately identify cropland in African countries. This study provides a quantitative evaluation and intercomparison of 11 publicly available land cover maps to assess their suitability for cropland classification and EO-based agriculture monitoring in Africa using statistically rigorous reference datasets from 8 countries. We hope the results of this study will help users determine the most suitable map for their needs and encourage future work to focus on resolving inconsistencies between maps and improving accuracy in low-accuracy regions.
BibTeX
@inproceedings{kerner2024how,
  author = {Hannah Kerner and Catherine Nakalembe and Adam Yang and Ivan Zvonkov and R. McWeeny and Gabriel Tseng and Inbal Becker‐Reshef},
  title = {{How accurate are existing land cover maps for agriculture in Sub-Saharan Africa?}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41597-024-03306-z},
}
Figure