Combining Deep Learning and Street View Imagery to Map Smallholder Crop Types

Jordi Laguarta Soler, Thomas Friedel, Sherrie WangAAAI 2024 · DOI

TL;DR

This paper introduces an automated pipeline to generate crop type maps for smallholder farming regions, overcoming the prohibitive cost of traditional ground surveys. By leveraging Google Street View (GSV) imagery and deep learning, they create a detailed crop map of Thailand with 93% accuracy, a significant advancement for regions where such data is scarce.

What Problem Does This Paper Address?

Accurate crop type maps are crucial for global food security, yield monitoring, and policy planning. However, creating these maps is challenging and expensive, especially in low- and middle-income countries where smallholder farms dominate. Traditional methods rely on costly and time-consuming field surveys, which are often infeasible for these regions. While remote sensing offers a scalable solution, it requires accurate ground truth data for training machine learning models, which is precisely the bottleneck. This paper aims to bridge this gap by using readily available street-level imagery to automatically generate these much-needed ground references.

Methods

The authors propose a multi-stage pipeline:

  1. Automated Field Point Generation: They start with OpenStreetMap (OSM) data to generate millions of equidistant points along road networks across Thailand. These points are then filtered using land cover maps (WorldCover) to identify areas likely to be cropland and exclude areas with significant tree cover that would obstruct views.
  2. GSV Image Filtering and Classification: For the filtered points, they extract Google Street View (GSV) images. A crucial step is filtering these images to ensure they actually show crop fields, not just roadside vegetation or other obstructions. They train a binary classifier (field vs. not field) for this purpose.
  3. Weakly Supervised Crop Type Classification: To label the crop types in the filtered GSV images, they explore several strategies:
  4. Web-based Datasets (WebCC): Images scraped from Google Images and iNaturalist, labeled by their search query or user tags.
  5. AI-Assisted Labeling (GPT-4V): Utilizing GPT-4V's zero-shot capabilities to label GSV images.
  6. Expert-Labeled Data: A smaller, high-quality dataset manually labeled by a plant taxonomy expert. They train a Convolutional Neural Network (CNN) on these datasets to classify crop types in street-view images. A "Mode of High Probabilities" (MHP) approach is used to aggregate predictions from sliding windows within an image for a more robust final classification.

  7. Remote Sensing-Based Crop Type Mapping: The crop type labels generated from GSV images (acting as ground references) are then used to train a Random Forest model on Sentinel-2 satellite time-series data. Harmonic regression is employed for feature extraction from the satellite imagery. This final model produces a wall-to-wall crop type map.

Datasets

  • OpenStreetMap (OSM): Used for road network data to generate initial points.
  • WorldCover 10m v100: A global land cover map used for filtering cropland.
  • Google Street View (GSV): The primary source of street-level imagery. Millions of images were collected and filtered.
  • WebCC (Web Crawled Content): Images scraped from Google Images, labeled by search queries.
  • iNaturalist: Images from a biodiversity database, used for crop type classification.
  • Expert-Labeled GSV Dataset: A smaller, manually curated dataset of GSV images labeled by a plant taxonomy expert, used for evaluation and training high-performance models.
  • Sentinel-2 Satellite Time Series: Used for the final crop type mapping.

The paper introduces a large dataset of 81,000 automatically generated ground reference points for Thailand.

Key Insights and Ideas

  • Street View as a Scalable Ground Truth Source: The core insight is that GSV, despite its limitations (occlusions, varying quality), can be effectively processed to generate a large volume of crop type ground references, significantly reducing the reliance on expensive manual surveys.
  • Weak Supervision is Powerful, but Noise Matters: While weakly supervised methods using web data (WebCC, iNaturalist) can train decent classifiers, they introduce noise. Combining these with expert labels or using more advanced AI (GPT-4V) yields better results.
  • Volume Trumps Perfection (to an extent): The study demonstrates that a large number of noisy ground references (81,000) generated automatically can lead to a more accurate final crop map than a much smaller set of perfectly labeled expert data (1,153).
  • GPT-4V as a Cost-Effective Labeling Tool: GPT-4V shows promising zero-shot performance for labeling GSV images, offering a potential pathway to further reduce manual labeling costs.
  • Smallholder Landscapes are Complex: The filtering steps (cropland, tree cover, field/not field classification) are critical for success in complex smallholder environments, unlike simpler agricultural landscapes.

Key Results

  • High Accuracy Crop Map: The generated national crop type map for Thailand achieved 93% accuracy, validated on 1,600 fields labeled by agronomists.
  • Cost-Effectiveness: The automated pipeline is orders of magnitude cheaper than traditional survey methods.
  • GSV Image Filtering Efficacy: The pipeline achieved a 98% precision and 95% recall in identifying field images after all filtering steps.
  • Street-View Classifier Performance: Models trained on combined datasets (iNaturalist + WebCC) achieved 85% overall accuracy, while models trained on expert or GPT-4V labeled data reached 92-93% accuracy.
  • Remote Sensing Model Performance: The final crop type map achieved an overall accuracy of 0.93 and an F1 score of 0.92. Rice and sugarcane were classified with high F1 scores (around 95% and 90-91% respectively), while maize and cassava were more challenging.
  • Scalability: The method generated 81,000 ground reference points, significantly more than what could be achieved through manual labeling alone.

Limitations and Future Work

  • Street View Availability and Update Frequency: The availability and update cycle of GSV services vary globally, limiting its universal applicability as a sole solution.
  • Occlusions: While addressed, some level of occlusion from vegetation still poses a challenge.
  • GPT-4V Query Limits: Current limitations on GPT-4V queries make large-scale labeling difficult.
  • Domain Expertise for Field/Not Field Classification: While not requiring crop-specific knowledge, the initial field/not field classification still required manual labeling. The authors suggest GPT-4V could potentially automate this as well.
  • Future Work:
    • Exploring transfer learning of remote sensing models to non-GSV regions.
  • Investigating the transferability of models over time.
  • Further automating the field/not field classification.
  • Expanding the approach to other regions and crop types.
  • Leveraging newer GSV cameras and expanding coverage.

Summary updated

BibTeX
@inproceedings{soler2024combining,
  author = {Jordi Laguarta Soler and Thomas Friedel and Sherrie Wang},
  title = {{Combining Deep Learning and Street View Imagery to Map Smallholder Crop Types}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30225},
}
Figure