EasyDAM_V2: Efficient Data Labeling Method for Multishape, Cross-Species Fruit Detection

Wenli Zhang, Kaizhen Chen, Chao Zheng, Yu‐xin Liu, Wei GuoPlant 2022 · DOI

TL;DR

This paper introduces EasyDAM_V2, a novel method for efficiently labeling fruit detection datasets across different species and shapes. By combining an improved image translation network (Across-CycleGAN) with an adaptive pseudolabeling strategy, it significantly reduces the cost and effort of creating large, labeled datasets for smart orchards.

What Problem Does This Paper Address?

Training high-performance fruit detection models for automated orchard systems (like fruit positioning, yield estimation, and robotic picking) relies heavily on large, meticulously labeled datasets. However, creating these datasets is a time-consuming and expensive process. Furthermore, deep learning models often struggle with generalization, meaning new models and datasets are needed for different fruit species, origins, or even varying lighting conditions. This necessitates efficient methods for transferring labels from existing datasets to new, unlabeled ones, especially when dealing with variations in fruit shape and appearance. Previous attempts, like the authors' own EasyDAM, could transfer color and texture but struggled with shape differences, and relied on manual tuning of pseudolabeling thresholds, limiting their efficiency.

Methods

EasyDAM_V2 builds upon previous work by introducing two key improvements:

  • Across-CycleGAN: This is a novel unsupervised image translation network designed to handle fruit images with partial shape differences. It enhances the original CycleGAN by:
  • Multilayer Feature Fusion Strategy: Fusing features from different network depths to better capture global shape information and local color/texture details.
  • Multidimensional Feature Loss Function: Incorporating a specific "shape feature loss" alongside traditional cycle consistency, identity, and adversarial losses. This loss function uses MS-SSIM to explicitly encourage the network to learn and preserve shape characteristics.
  • Cross-Cycle Loss Function Comparison Path: This design allows for direct comparison of shape features between synthetic and actual images across different domain cycles, further improving shape translation.
  • Pseudolabel Adaptive Threshold Selection Strategy: This strategy automates the selection of the optimal confidence threshold for generating pseudolabels. Instead of manual tuning, it analyzes the quantity and score distribution of pseudolabels generated at various confidence thresholds. It identifies the threshold that maximizes the "quality variance" of the pseudolabels, effectively balancing the number of detected objects with their confidence scores to minimize noisy labels.

Datasets

The study utilized the following datasets:

  • Fruit Image Translation Datasets:
    • orange&pitaya: 980 orange images (source) and 416 pitaya images (target).
  • orange&mango: 980 orange images (source) and 141 mango images (target).
  • Additional target domain fruits (pear, kiwi, green pepper) were used for broader validation.
  • Fruit Detection Datasets:
    • Source Domain: 664 labeled orange images collected from an orchard in Sichuan, China.
  • Target Domain: * Pitaya: 377 images (265 training, 112 testing) collected from an orchard in Beijing, China, and online.
  • Mango: 620 images (516 training, 104 testing) from a published dataset collected in Australia.

Key Insights and Ideas

  • Shape Matters: The paper strongly emphasizes that for effective cross-species fruit detection, accurately translating shape features is as crucial as color and texture. The Across-CycleGAN's explicit shape loss function is a direct response to this.
  • Automating Pseudolabeling: Manually tuning pseudolabel confidence thresholds is a significant bottleneck. The adaptive threshold selection strategy offers a data-driven, automated solution that improves efficiency and potentially accuracy.
  • Bridging the Gap: The combination of improved image translation and adaptive pseudolabeling creates a powerful pipeline for transferring labels even when significant shape differences exist between source and target fruit species.
  • Real-world Applicability: The method is designed to address practical challenges in smart agriculture, aiming to reduce the cost and labor associated with dataset creation for fruit detection.

Key Results

  • Image Translation Performance: The Across-CycleGAN demonstrated superior performance in generating synthetic pitaya and mango images compared to U-GAT-IT and Council-GAN. For pitaya detection models, it achieved an AP of 72.8% and an F1 score of 71.0%, outperforming U-GAT-IT by 5.3% (AP) and Council-GAN by 61.8% (AP). For mango detection, it achieved an AP of 68.7% and an F1 score of 67.8%.
  • Pseudolabeling Accuracy:
    • For pitaya, the proposed method achieved an AP of 82.1% and an F1 score of 78.0%, which was 3.9% higher than the best traditional pseudolabeling method (T-PL).
  • For mango, the proposed method achieved an AP of 85.0% and an F1 score of 81.7%, outperforming T-PL by 1.2% (AP) and PL-SL by 0.5% (AP).
  • Qualitative Results: The generated synthetic images, even for fruits with significant shape differences (e.g., oranges to cucumbers/eggplants), showed improved shape translation compared to previous methods. The method also demonstrated robustness to differences in scene conditions (indoor vs. outdoor).

Limitations and Future Work

  • Large Shape Differences: The authors acknowledge that while Across-CycleGAN handles partial shape differences well, it requires further improvement for fruit images with large shape differences (e.g., orange to cucumber). The synthetic cucumbers in their example did not align positionally with the oranges, and eggplants were not generated one-to-one.
  • Loss Function Weighting: The optimal weights for the different loss functions in the Across-CycleGAN are currently set empirically. Future work could involve automatically assigning these weights based on feature differences between fruit images.
  • Mislabeling due to Scale Differences: Differences in fruit scale between synthetic and actual images can lead to mislabeling during pseudolabel generation. Future research aims to address this.
  • Further Filtering of Noisy Labels: While the adaptive threshold strategy reduces noisy labels, additional filtering might be beneficial for even higher precision.
  • Generalizability: The paper focuses on fruit detection. Extending the approach to other object detection tasks or more complex scenarios could be explored.

Summary updated

Abstract

In modern smart orchards, fruit detection models based on deep learning require expensive dataset labeling work to support the construction of detection models, resulting in high model application costs. Our previous work combined generative adversarial networks (GANs) and pseudolabeling methods to transfer labels from one specie to another to save labeling costs. However, only the color and texture features of images can be migrated, which still needs improvement in the accuracy of the data labeling. Therefore, this study proposes an EasyDAM_V2 model as an improved data labeling method for multishape and cross-species fruit detection. First, an image translation network named the Across-CycleGAN is proposed to generate fruit images from the source domain (fruit image with labels) to the target domain (fruit image without labels) even with partial shape differences. Then, a pseudolabel adaptive threshold selection strategy was designed to adjust the confidence threshold of the fruit detection model adaptively and dynamically update the pseudolabel to generate labels for images from the unlabeled target domain. In this paper, we use a labeled orange dataset as the source domain, and a pitaya, a mango dataset as the target domain, to evaluate the performance of the proposed method. The results showed that the average labeling precision values of the pitaya and mango datasets were 82.1% and 85.0%, respectively. Therefore, the proposed EasyDAM_V2 model is proven to be used for label transfer of cross-species fruit even with partial shape differences to reduce the cost of data labeling.
BibTeX
@inproceedings{zhang2022easydam,
  author = {Wenli Zhang and Kaizhen Chen and Chao Zheng and Yu‐xin Liu and Wei Guo},
  title = {{EasyDAM\_V2: Efficient Data Labeling Method for Multishape, Cross-Species Fruit Detection}},
  booktitle = {Plant},
  year = {2022},
  doi = {10.34133/2022/9761674},
}
Figure