A Weakly Supervised Semantic Segmentation Model of Maize Seedlings and Weed Images Based on Scrawl Labels

Lulu Zhao, Yanan Zhao, Ting Liu, Hanbing DengSensors 2023 · OpenReview

TL;DR

This paper introduces SL-Net, a novel weakly supervised semantic segmentation model for identifying maize seedlings and weeds. By leveraging "scrawl" labels (simple line drawings) and a clever pseudo-label generation module, SL-Net significantly reduces the annotation burden compared to traditional pixel-level labeling, achieving competitive accuracy with fully supervised methods.

What Problem Does This Paper Address?

Accurately segmenting crops and weeds is crucial for precision agriculture, enabling targeted weed removal and optimizing resource use. However, existing deep learning methods for semantic segmentation demand extensive pixel-level annotations (masks) for every image. Creating these masks is incredibly time-consuming and labor-intensive, especially for complex agricultural scenes with varying plant morphologies. This high annotation cost is a major bottleneck, hindering the widespread adoption of these powerful techniques. The paper aims to bridge this gap by developing a model that can learn effectively from less precise, "weak" labels.

Methods

The core innovation of SL-Net lies in its pseudo-label generation module. This module takes "scrawl" labels—simple line drawings indicating the presence of maize or weeds—and transforms them into more informative pseudo-pixel-level labels. The process involves:

  • Feature Extraction: Using Exceed Green feature Graying (EXG) to highlight green areas.
  • Initial Masking: Employing GrabCut for initial maize mask generation and Otsu's method for weed detection.
  • Refinement: Utilizing a combination of morphological operations (corrosion) and DenseCRF (Dense Conditional Random Fields) to refine the boundaries and improve the accuracy of the generated pseudo-labels.

The encoder-decoder architecture of SL-Net is based on DeepLab-V3+, but with a key modification: it replaces the heavier Xception backbone with the more efficient MobileNet-V2. This choice significantly reduces the number of parameters and computational cost while maintaining high feature extraction capabilities. The model also incorporates an Atrous Spatial Pyramid Pooling (ASPP) module for capturing multi-scale contextual information.

For training, SL-Net utilizes a combined loss function comprising balanced cross-entropy loss (LCE) and Dice loss (LDic). This combination helps focus the model on learning foreground objects and provides a robust similarity measure. The paper also mentions the use of migration learning to leverage pre-trained weights and optimize the training process.

Datasets

The study uses a custom dataset of maize seedling and weed images.

  • Data Acquisition: Images were captured from a top-down view of maize plants at the seedling stage using a conventional optical camera.
  • Preprocessing: Images were standardized to 1024x1024 pixels. Data augmentation techniques like mirror flipping, rotation, and pretzel noise were applied to enhance generalization.
  • Dataset Split: The final dataset consisted of 1200 images, split into training (960 images), testing (120 images), and validation (120 images) sets.
  • Labeling: The primary labels used for training were "scrawl" labels, which are line drawings indicating maize and weed areas. These were then converted into pseudo-pixel-level labels. Ground truth pixel-level masks were also used for evaluation.

Key Insights and Ideas

  • Scrawl Labels as a Viable Alternative: The research demonstrates that simple scrawl labels, when processed effectively, can provide sufficient information for training accurate semantic segmentation models, drastically reducing annotation costs.
  • Pseudo-Label Quality Matters: The effectiveness of the scrawl-label approach hinges on the quality of the generated pseudo-labels. The paper's multi-stage refinement process (GrabCut, DenseCRF, corrosion) is crucial for achieving high-quality pseudo-labels that closely resemble ground truth.
  • MobileNet-V2 for Efficiency: Replacing the Xception backbone with MobileNet-V2 in a DeepLab-V3+ architecture offers a significant advantage in terms of computational efficiency without a substantial drop in performance, making the model more practical for real-world applications.
  • Weak Supervision Can Compete: The study shows that a well-designed weakly supervised model can achieve performance comparable to, and in some cases even better than, fully supervised models, challenging the notion that pixel-level annotations are always necessary for top-tier results.

Key Results

  • Pseudo-Label Quality: The generated pseudo-labels achieved a Mean Intersection over Union (MIoU) of 83.32% and a cosine similarity of 93.55% with the ground truth, indicating high fidelity.
  • Weakly Supervised Performance: SL-Net achieved a mean Intersection over Union (MIoU) of 87.30% and a Mean Pixel Accuracy (MPA) of 94.06% on the maize and weed segmentation task.
  • Comparison with Baselines (Weakly Supervised):
    • SL-Net outperformed the original DeepLab-V3+ by 3.48% in MIoU and the original PSPNet by 8.25% in MIoU.
  • When comparing against improved versions of the baselines (using MobileNet-V2 and combined loss), SL-Net still showed superior performance, achieving 0.11% higher MIoU than the best DeepLab-V3+ variant and 8.25% higher MIoU than the best PSPNet variant.
  • Comparison with Fully Supervised Models: SL-Net's accuracy was comparable to or better than several fully supervised models, reaching up to 109.02% of the accuracy of the PSPNet model (with MobileNet-V2 backbone).
  • Detection Speed: SL-Net achieved a detection speed of 21.2 frames/s, which is significantly faster than the original DeepLab-V3+ (7.52 frames/s) and comparable to other optimized models.

Limitations and Future Work

  • Dataset Specificity: The study focused on maize seedlings and weeds. The generalizability of SL-Net to other crop types or weed species might require further investigation.
  • Scrawl Label Ambiguity: While scrawl labels reduce annotation effort, there's a potential for ambiguity or inconsistency in how users draw them, which could impact pseudo-label quality. The paper mentions that "large areas of nongreen plants" could affect pseudo-label quality, suggesting a limitation in handling diverse background conditions.
  • Future Work: The authors suggest that SL-Net provides a strong foundation for further research in agricultural image analysis, particularly in weed segmentation and phenotype analysis, aiming for even higher precision and lower costs. The paper implicitly suggests exploring more sophisticated pseudo-labeling techniques or adapting the model to a wider range of agricultural scenarios.

Summary updated

Abstract

The task of semantic segmentation of maize and weed images using fully supervised deep learning models requires a large number of pixel-level mask labels, and the complex morphology of the maize and weeds themselves can further increase the cost of image annotation. To solve this problem, we proposed a Scrawl Label-based Weakly Supervised Semantic Segmentation Network (SL-Net). SL-Net consists of a pseudo label generation module, encoder, and decoder. The pseudo label generation module converts scrawl labels into pseudo labels that replace manual labels that are involved in network training, improving the backbone network for feature extraction based on the DeepLab-V3+ model and using a migration learning strategy to optimize the training process. The results show that the intersection over union of the pseudo labels that are generated by the pseudo label module with the ground truth is 83.32%, and the cosine similarity is 93.55%. In the semantic segmentation testing of SL-Net for image seedling of maize plants and weeds, the mean intersection over union and average precision reached 87.30% and 94.06%, which is higher than the semantic segmentation accuracy of DeepLab-V3+ and PSPNet under weakly and fully supervised learning conditions. We conduct experiments to demonstrate the effectiveness of the proposed method.
BibTeX
@inproceedings{zhao2023weakly,
  author = {Lulu Zhao and Yanan Zhao and Ting Liu and Hanbing Deng},
  title = {{A Weakly Supervised Semantic Segmentation Model of Maize Seedlings and Weed Images Based on Scrawl Labels}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=qWj3kad5Mi},
}
Figure