EasyDAM_V3: Automatic Fruit Labeling Based on Optimal Source Domain Selection and Data Synthesis via a Knowledge Graph

Wenli Zhang, Yuxin Liu, Chao Zheng, Guoqiang Cui, Wei GuoPlant 2023 · DOI

TL;DR

This paper introduces EasyDAM_V3, an automated fruit labeling system that significantly reduces the need for manual annotation. By intelligently selecting optimal "source" fruit datasets and synthesizing realistic "target" domain images using a knowledge graph, it achieves high accuracy in fruit detection and labeling, paving the way for more efficient smart agriculture.

What Problem Does This Paper Address?

Deep learning models are revolutionizing agriculture, particularly in areas like smart orchards for tasks such as fruit detection, sorting, and picking. However, these models are data-hungry, requiring vast amounts of labeled training data. Manually labeling fruit images is a tedious, time-consuming, and expensive process. Previous automated labeling methods (EasyDAM_V1 and V2) helped reduce this burden but had limitations: they often relied on subjective source domain selection and still required some manual effort for source data preparation. Furthermore, their performance could fluctuate when dealing with diverse fruit types and complex orchard environments. This paper aims to overcome these limitations by developing a fully automated system that can select the best source data and generate realistic synthetic data for training, thereby eliminating manual labeling costs and improving domain adaptability.

Methods

EasyDAM_V3 builds upon previous work by introducing two key innovations:

  1. Optimal Source Domain Selection: Instead of subjective choices, this method uses a multidimensional spatial feature model to objectively select the most suitable "source" fruit dataset. It quantifies phenotypic characteristics (shape, color, texture) of various fruits, fuses them into a multidimensional feature space, and employs density-based spatial clustering to identify the optimal source domain that best represents multiple target fruit types. This ensures better domain adaptation for image translation.
  2. Knowledge Graph-Based Data Synthesis: To create realistic training data for target domains, the paper proposes a novel approach using a knowledge graph. This graph encodes hierarchical rules for synthesizing orchard scenes, including natural semantic composition, growth semantics, and environmental adaptation. By combining transparent background fruit images (generated via CycleGAN translation from the optimal source domain) with these rules, the system generates high-volume synthetic datasets that mimic real-world orchard complexities (occlusion, lighting, scale variations).

These synthesized datasets are then used to train an anchor-free fruit detection model (CenterNet), which is further refined using a pseudo-label self-learning strategy to achieve accurate fruit labeling.

Datasets

The study utilizes two main types of datasets:

  • Transparent Background Fruit Dataset: This dataset comprises 14 fruit types (e.g., apple, banana, citrus, pear, tomato) with 202 images each, used for phenotypic characterization and optimal source domain selection.
  • Actual Orchard Fruit Dataset: This dataset consists of real-world images of citrus, apple, and tomato collected from orchards. These are used for evaluating the performance of the EasyDAM_V3 model.

The paper also mentions using publicly available datasets and web crawling for data acquisition. The generated synthetic datasets are a key output of the proposed method.

Key Insights and Ideas

  • Objective Source Domain Selection is Crucial: The paper demonstrates that moving from subjective to objective, feature-based selection of source domains significantly improves the robustness and generalizability of image translation models for fruit labeling.
  • Knowledge Graphs for Realistic Synthesis: Leveraging knowledge graphs to encode hierarchical rules for scene composition allows for the generation of highly realistic synthetic orchard images, overcoming the limitations of simple background replacement.
  • Bridging Domain Gaps: The combination of optimal source domain selection and sophisticated data synthesis effectively bridges the domain gap between source and target datasets, leading to more accurate and reliable fruit detection.
  • Zero-Cost Labeling is Achievable: The proposed end-to-end automated pipeline successfully eliminates the need for manual labeling of source or target datasets, achieving the goal of zero-cost automated fruit labeling.

Key Results

The EasyDAM_V3 model achieved impressive results in automatic fruit labeling:

  • High Average Precision: The average precision (mAP) for the target datasets (orange, apple, and tomato) reached 90.94%, 89.78%, and 90.84%, respectively.
  • Improved Performance: Compared to the previous EasyDAM_V1 method, EasyDAM_V3 showed significant improvements in precision, recall, and average precision, particularly for the pear2apple and pear2tomato experiments.
  • Robustness: The proposed method demonstrated improved performance and better generalization across different angles, shooting distances, and environments, even with large variations in fruit size and scene complexity.
  • Efficiency: By reducing the need for manual labeling and extensive data collection, the method significantly lowers associated costs and labor.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that for fruits with very rich appearance features and discrete distributions (e.g., pitaya, cucumber, banana), selecting a single optimal source domain might be challenging, requiring further research into multi-source domain strategies.
  • The phenotypic characterization of the 14 fruit datasets was limited due to labor costs and seasonal constraints.
  • Future Work:
    • Improve synthetic dataset construction by incorporating depth information and 3D model reconstruction for even greater realism.
  • Further enhance fruit growth subrules and branch/leaf colonization rules to improve the intelligence and accuracy of the automatic labeling method.
  • Conduct more detailed analysis of different fruits within the same species to capture intra-species variations.
  • Explore multi-source domain strategies for complex fruit types.

Summary updated

Abstract

Although deep learning-based fruit detection techniques are becoming popular, they require a large number of labeled datasets to support model training. Moreover, the manual labeling process is time-consuming and labor-intensive. We previously implemented a generative adversarial network-based method to reduce labeling costs. However, it does not consider fitness among more species. Methods of selecting the most suitable source domain dataset based on the fruit datasets of the target domain remain to be investigated. Moreover, current automatic labeling technology still requires manual labeling of the source domain dataset and cannot completely eliminate manual processes. Therefore, an improved EasyDAM_V3 model was proposed in this study as an automatic labeling method for additional classes of fruit. This study proposes both an optimal source domain establishment method based on a multidimensional spatial feature model to select the most suitable source domain, and a high-volume dataset construction method based on transparent background fruit image translation by constructing a knowledge graph of orchard scene hierarchy component synthesis rules. The EasyDAM_V3 model can automatically obtain fruit label information from the dataset, thereby eliminating manual labeling. To test the proposed method, pear was used as the selected optimal source domain, followed by orange, apple, and tomato as the target domain datasets. The results showed that the average precision of annotation reached 90.94%, 89.78%, and 90.84% for the target datasets, respectively. The EasyDAM_V3 model can obtain the optimal source domain in automatic labeling tasks, thus eliminating the manual labeling process and reducing associated costs and labor.
BibTeX
@inproceedings{zhang2023easydam,
  author = {Wenli Zhang and Yuxin Liu and Chao Zheng and Guoqiang Cui and Wei Guo},
  title = {{EasyDAM\_V3: Automatic Fruit Labeling Based on Optimal Source Domain Selection and Data Synthesis via a Knowledge Graph}},
  booktitle = {Plant},
  year = {2023},
  doi = {10.34133/plantphenomics.0067},
}
Figure