The CropAndWeed Dataset: A Multi-Modal Learning Approach for Efficient Crop and Weed Manipulation

Daniel Steininger, Andreas Trondl, Gerardus Croonen, Julia Simon, Verena WidhalmWACV 2023

TL;DR

This paper introduces the CropAndWeed dataset, a large-scale, multi-modal dataset for precision agriculture, featuring 74 crop and weed species with rich annotations. They demonstrate its utility by training various models and showing that incorporating weed species significantly improves crop detection accuracy, addressing a critical data gap in the field.

What Problem Does This Paper Address?

Precision agriculture, particularly automated weed detection and removal, is crucial for sustainable food production. While deep learning methods show great promise, their effectiveness hinges on large, diverse, and accurately annotated datasets. Existing datasets in this domain are often limited in the number of species, data variability (e.g., lighting, soil conditions), and annotation types (e.g., lacking semantic masks or stem positions). This lack of comprehensive data hinders the development of robust and generalized AI systems for efficient crop and weed management.

Methods

The authors propose a novel, large-scale dataset called CropAndWeed. The dataset is built upon a multi-modal approach, incorporating RGB images with extensive annotations. These annotations include:

  • Bounding Boxes: For object detection.
  • Semantic Masks: For pixel-level segmentation of plants.
  • Stem Positions: Crucial for precise plant localization.
  • Fine-grained Classification: Differentiating between 16 crop and 58 weed species.
  • Meta-annotations: Including environmental conditions (lighting, soil type, moisture) and recording parameters, which are vital for understanding data variability and evaluating model robustness.

To demonstrate the dataset's versatility, they conduct benchmark experiments for object detection, stem localization, and segmentation. They also introduce a concept of "data ablation," where they create various dataset variants by mapping the original 74 classes to different super-classes. This allows them to explore how different levels of granularity and the inclusion of weed species impact model performance.

Datasets

The primary contribution is the CropAndWeed dataset, which comprises:

  • Over 8,000 high-resolution images.
  • Approximately 112,000 annotated instances.
  • 74 distinct plant classes (16 crops, 58 weeds).
  • Two main subsets: an "Application Set" from real-world agricultural sites and an "Experimental Set" from controlled outdoor plots for rare weed species.

The dataset is split into training, validation, and testing sets with a 70:15:15 ratio. They also define several dataset variants for evaluation, including:

  • Fine24: A fine-grained variant with 8 crop and 16 weed classes.
  • CropsOrWeed9: A variant with 9 crop classes and a single weed class.
  • Crop2: A variant with 2 classes (specific crop + all other plants).
  • Crop1: Variants trained on individual crop classes.
  • Coarsel: A binary classification of soil vs. vegetation.

Key Insights and Ideas

  • Weeds are Crucial for Crop Detection: A key finding is that incorporating weed species into the training process significantly improves the accuracy of crop detection, even when the ultimate goal is to identify only crops. This challenges the common approach of focusing solely on crop classes.
  • Data Variability is Key: The dataset's emphasis on diverse environmental conditions (lighting, soil, weather) and recording parameters is crucial for developing models that generalize well to real-world agricultural scenarios.
  • Multi-modal Annotations Enable Versatility: The combination of bounding boxes, semantic masks, and stem positions allows for a wide range of downstream tasks and enables the integration of different learning modules (e.g., detection with segmentation).
  • Data Ablation for Customization: The concept of creating dataset variants allows researchers to tailor the dataset to specific application needs, focusing on the relevant crop and weed species for a particular region or farm.

Key Results

  • Detection Performance: The CropAndWeed dataset enables strong detection performance. For instance, on the Fine24 variant, models achieved an overall AP of 55.4%. Performance generally increased with instance size, with small instances (162-322 pixels) already showing promising robustness.
  • Impact of Weed Inclusion: Models trained with weed classes included (e.g., CropsOrWeed9) consistently outperformed models trained on single crop classes (Crop1) or only crops and a general weed class (Crop2). For example, CropsOrWeed9 achieved an AP of 71.5%, significantly higher than Crop1's 57.3%.
  • Segmentation Performance: Segmentation models achieved high mIoU scores, particularly for the Coarsel variant (87.7%), which distinguishes between soil and vegetation. Per-class IoU for specific crops was also respectable, with Maize reaching 76.5% on the Fine24 variant.
  • Stem Localization: The stem localization module achieved a promising detection rate of 73.1% with a mean Euclidean distance of 29.1 pixels.

Limitations and Future Work

  • Data Scarcity for Rare Weeds: While the dataset includes rare weed species, some might still be underrepresented for training highly specialized models. The authors suggest further enriching the dataset with more samples.
  • Limited Modalities: The current dataset primarily uses RGB images. Future work could incorporate additional modalities like multispectral images or data from UAVs for enhanced efficiency and broader applicability.
  • Object Tracking: The authors plan to extend the dataset by adding object tracking capabilities to re-identify individual plants over time, further enhancing robustness.
  • Scalability of Annotations: The manual annotation process, while thorough, is labor-intensive. Future efforts might explore semi-automated or active learning approaches to scale annotation efforts.

Summary updated

BibTeX
@inproceedings{steininger2023cropandweed,
  author = {Daniel Steininger and Andreas Trondl and Gerardus Croonen and Julia Simon and Verena Widhalm},
  title = {{The CropAndWeed Dataset: A Multi-Modal Learning Approach for Efficient Crop and Weed Manipulation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure