SoybeanNet: Transformer-based convolutional neural network for soybean pod counting from Unmanned Aerial Vehicle (UAV) images

Xiang Yin (殷翔), Jiajia Li, Raju Thada Magar, Dong Chen, Feng Lin, Dechun Wang, Xiang Yin, Weichao Zhuang, Zhaojian Li

TL;DR

This paper introduces SoybeanNet, a novel transformer-based deep learning model for accurately counting soybean pods directly from UAV imagery. They also release a new, large-scale dataset of annotated soybean pods captured in real-world field conditions, addressing a critical need for robust agricultural AI tools.

What Problem Does This Paper Address?

Accurate soybean yield prediction is crucial for global food security and efficient agricultural practices. A key indicator of yield is the number of pods per plant. However, manually counting these pods in large fields is incredibly labor-intensive and time-consuming, especially for plant breeders working with numerous soybean lines. Existing automated methods often rely on controlled lab environments or ground-based vehicles, which have limitations in terms of practicality and scalability. Furthermore, many methods require costly bounding box annotations, which are often unnecessary for simple counting tasks. This paper aims to bridge this gap by developing a robust, automated soybean pod counting system using readily available UAV imagery and a more efficient annotation strategy.

Methods

The core of this work is SoybeanNet, a novel deep learning architecture that combines the power of transformer networks with convolutional neural networks. SoybeanNet employs a transformer-based backbone (specifically, variants of Swin Transformers) to extract rich, hierarchical features from the UAV images. These features are then processed through a "neck" component, inspired by UNet, which upsamples and merges feature maps to enhance spatial resolution and capture multi-scale information. Finally, a "head" component with two branches predicts point proposals for soybean pods: one branch for coordinate regression (pinpointing the pod's location) and another for classification (estimating the confidence score of each proposal).

A key innovation is the use of a point-based counting approach with dot annotations (marking the center of each pod) rather than bounding boxes. To ensure accurate matching between predicted points and ground truth, they implement a one-to-one matching strategy using the Hungarian algorithm. This approach guarantees that each pod is counted only once and helps prevent over- or under-counting. The model is trained using a composite loss function that combines Euclidean loss for localization and cross-entropy loss for classification.

Datasets

The paper introduces a newly created and open-sourced UAV-acquired soybean dataset. This dataset consists of:

  • 113 drone images captured during the full maturation stage in natural field conditions in Michigan, USA.
  • Over 260,000 manually annotated soybean pods using dot annotations.
  • Images were collected using a DJI Mavic2 UAV at an altitude of around 13 feet, with varying resolutions from 800x800 to 2000x2000.
  • The dataset includes images from two different soybean fields and two distinct soybean varieties (Tan and Brown).

This dataset is partitioned into training (70%), validation (15%), and testing (15%) subsets.

Key Insights and Ideas

  • Transformer Power for Agricultural Tasks: The paper demonstrates that transformer architectures, particularly Swin Transformers, are highly effective for extracting features from complex agricultural imagery, outperforming previous CNN-based approaches.
  • Efficiency of Point-Based Counting: Using dot annotations and a point-proposal network significantly reduces the annotation burden compared to bounding boxes, making data collection more scalable.
  • Real-World Robustness: The model's ability to handle variations in lighting, soil background, plant density, and pod overlap in natural field conditions is a significant step towards practical agricultural AI.
  • Complexity vs. Performance: Interestingly, the largest model variant (SoybeanNet-L) did not achieve the absolute best performance, suggesting that model complexity alone doesn't guarantee superior results and highlighting the importance of architecture design and training.

Key Results

SoybeanNet demonstrated superior performance compared to five state-of-the-art crowd-counting and plant-counting methods on their newly created dataset.

  • Accuracy: SoybeanNet achieved a counting accuracy of 84.51% on the test dataset, with specific variants like SoybeanNet-S achieving the best R2 value of 0.8642.
  • MAE: The proposed approach achieved a Mean Absolute Error (MAE) generally less than 370, outperforming benchmarks like DM-Count and P2PNet.
  • Correlation with Yield: A preliminary study showed a strong positive correlation (r=0.6618) between the predicted pod number and actual yield, validating the utility of pod counting for yield prediction.
  • Qualitative Results: Visualizations (Figure 3) show the model effectively predicting pod centers even in dense and overlapping scenarios across different soybean varieties.

Limitations and Future Work

  • Resource-Intensive Annotation: The manual annotation process, while crucial, is time-consuming and costly. The authors suggest exploring label-efficient learning techniques to mitigate this.
  • Real-Time Deployment: The current evaluation is offline. Future work should focus on adapting the model for real-time inference, potentially through optimization techniques like model quantization and hardware acceleration.
  • Data Diversity: While the dataset is valuable, further research could explore more diverse agricultural environments and soybean varieties to enhance model generalization.
  • Yield Prediction Integration: While a preliminary correlation study was conducted, a more integrated approach to directly predict yield from UAV imagery, incorporating pod count as a feature, could be explored.

Summary updated

Abstract

Soybeans are a critical source of food, protein and oil, and thus have received extensive research aimed at enhancing their yield, refining cultivation practices, and advancing soybean breeding techniques. Within this context, soybean pod counting plays an essential role in understanding and optimizing production. Despite recent advancements, the development of a robust pod-counting algorithm capable of performing effectively in real-field conditions remains a significant challenge This paper presents a pioneering work of accurate soybean pod counting utilizing unmanned aerial vehicle (UAV) images captured from actual soybean fields in Michigan, USA. Specifically, this paper presents SoybeanNet, a novel point-based counting network that harnesses powerful transformer backbones for simultaneous soybean pod counting and localization with high accuracy. In addition, a new dataset of UAV-acquired images for soybean pod counting was created and open-sourced, consisting of 113 drone images with more than 260k manually annotated soybean pods captured under natural lighting conditions. Through comprehensive evaluations, SoybeanNet demonstrated superior performance over five state-of-the-art approaches when tested on the collected images. Remarkably, SoybeanNet achieved a counting accuracy of $84.51\%$ when tested on the testing dataset, attesting to its efficacy in real-world scenarios. The publication also provides both the source code (\url{https://github.com/JiajiaLi04/Soybean-Pod-Counting-from-UAV-Images}) and the labeled soybean dataset (\url{https://www.kaggle.com/datasets/jiajiali/uav-based-soybean-pod-images}), offering a valuable resource for future research endeavors in soybean pod counting and related fields.
BibTeX
@article{2024soybeannet,
  author = {Xiang Yin (殷翔) and Jiajia Li and Raju Thada Magar and Dong Chen and Feng Lin and Dechun Wang and Xiang Yin and Weichao Zhuang and Zhaojian Li},
  title = {{SoybeanNet: Transformer-based convolutional neural network for soybean pod counting from Unmanned Aerial Vehicle (UAV) images}},
  journal = {Comput. Electron. Agric},
  year = {2024},
  doi = {10.1016/j.compag.2024.108861},
}
Figure