HarvestNet: A Dataset for Detecting Smallholder Farming Activity Using Harvest Piles and Remote Sensing

Jonathan Xu, Amna Elmustafa, Liya Weldegebriel, Emnet Negash, Richard Lee, Chenlin Meng, Stefano Ermon, David B. Lobell

Abstract

Accurate mapping of smallholder farms is essential for global food security, yet traditional remote sensing methods often fail in non-mechanized agricultural systems due to "green-on-green" confusion between crops and wild vegetation. We present HarvestNet, a novel dataset and detection framework that shifts the focus from spectral signatures of crops to the detection of "harvest piles"—temporary, human-made structures created during the harvest process. Using high-resolution SkySat imagery (0.5m), our models identified over 56,000 hectares of previously unmapped cropland in Ethiopia, significantly outperforming existing global land-cover products like ESA WorldCover.

Introduction: The Challenge of Smallholder Mapping

Smallholder farms, typically under two hectares, are the backbone of food production in sub-Saharan Africa. However, they are notoriously difficult to map using standard satellite imagery. Traditional methods rely on the "spectral signatures" of plants, but in many regions, weeds and wild grasses grow alongside crops, leading to significant classification errors. If agricultural activity cannot be accurately mapped, monitoring productivity and responding to food crises becomes nearly impossible.

A ground-level view of a harvest pile in Ethiopia, demonstrating the scale of these structures relative to a human.

Methodology: Detecting Human Activity

To overcome the limitations of vegetation-based mapping, HarvestNet focuses on harvest piles. These are stacks of crops, typically 3–10 meters in diameter, gathered by farmers for threshing at the end of the growing season. Unlike static field boundaries, these piles are dynamic, direct indicators of active seasonal farming.

The researchers developed a multi-tiered labeling pipeline to create a high-quality training set:

  1. Expert Distillation: Domain experts trained coordinators to distinguish harvest piles from similar-looking objects like aluminum sheds or dirt mounds.
  2. Crowdsourced Filtering: Amazon Mechanical Turk workers performed initial filtering of negative samples.
  3. Expert Verification: A final consensus check was performed by researchers to ensure label accuracy.
Satellite imagery from the HarvestNet dataset showing harvest piles (circled in red) across various agricultural landscapes in Ethiopia.

The HarvestNet Dataset

The dataset focuses on the Tigray and Amhara regions of Ethiopia (2020–2023) and includes:

  • 7,000 hand-labeled SkySat images at 0.5m resolution.
  • 2,000 ground-truth labels collected by field teams, serving as a gold standard for validation.
  • PlanetScope imagery (4.77m resolution) to evaluate model performance across different spatial scales.

Results and Discussion

The study benchmarked several architectures, including ResNet-50 and modern vision transformers like the Swin Autoencoder.

Model Performance

The Swin Autoencoder achieved the highest overall performance on hand-labeled data with an accuracy of 80.87% (F1-score: 77.23). Remarkably, the ResNet-50 model demonstrated exceptional accuracy when tested against ground-truth points, reaching 98.68% in Amhara and 90.76% in Tigray. This suggests that convolutional neural networks (CNNs) remain highly effective for detecting small, compact objects where local pixel structure is paramount.

The Resolution Gap

A key finding was the sensitivity of detection to spatial resolution. At 4.77m resolution (PlanetScope), harvest piles become nearly indistinguishable from the background. This indicates that high-resolution imagery (sub-meter) is currently a prerequisite for this specific detection task.

Correcting Global Maps

The model's real-world impact was demonstrated by identifying 56,621 hectares of active cropland in Tigray that were incorrectly classified as "non-cropland" by the ESA WorldCover map. This highlights a massive undercounting of agricultural activity in existing global databases.

Conclusion and Future Work

HarvestNet demonstrates that focusing on human-made indicators of farming activity can revolutionize remote sensing for smallholder agriculture. While the current approach is limited by the cost of high-resolution data and a focus on binary detection (presence vs. absence), it provides a critical tool for improving food security monitoring. Future research will explore the use of time-series data and automated labeling tools like "Segment Anything" to further scale this approach.

Summary updated

BibTeX
@inproceedings{xu2024harvestnet,
  author = {Jonathan Xu and Amna Elmustafa and Liya Weldegebriel and Emnet Negash and Richard Lee and Chenlin Meng and Stefano Ermon and David B. Lobell},
  title = {{HarvestNet: A Dataset for Detecting Smallholder Farming Activity Using Harvest Piles and Remote Sensing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30251},
}
Figure