Extraction and Research of Crop Feature Points Based on Computer Vision

Jingwen Cui, Jianping Zhang, Guiling Sun, Bowen ZhengSensors 2019 · OpenReview

TL;DR

This paper presents a computer vision system for identifying and locating crops for automated harvesting. It combines the YOLOv3 object detection algorithm with point cloud image coordinate matching using a Kinect v2 depth camera, achieving high accuracy and low positioning error for robotic arm control.

What Problem Does This Paper Address?

Automated crop harvesting is a significant challenge in modern agriculture. A crucial step in this process is accurately identifying and locating crops in a 3D space so that robotic arms can effectively grasp them. Traditional object recognition methods can be complex and computationally intensive, and accurately obtaining 3D information from 2D images is difficult. This paper aims to bridge this gap by developing a robust and efficient method for crop feature point extraction and 3D localization, paving the way for more intelligent agricultural robots.

Methods

The core of this work lies in the innovative integration of two key technologies:

  • YOLOv3 Object Detection: The YOLOv3 algorithm, running on the DarkNet framework, is employed to identify and classify different types of target crops within RGB images. YOLOv3 is chosen for its speed and its ability to handle small and dense objects, which is common in agricultural settings.
  • Point Cloud Image Coordinate Matching: After identifying the crops with YOLOv3, the system uses a point cloud image coordinate matching method to determine the 3D coordinates of feature points on the identified crops. This is achieved by leveraging depth information from a Kinect v2 depth camera. The method involves:
  • Acquiring synchronized RGB and depth images from the Kinect v2.
  • Using YOLOv3 to detect crops in the RGB images and determine bounding boxes.
  • Defining feature points, typically at the center of the bounding box or diagonally linked corners, for each detected crop.
  • Mapping these 2D feature points to their corresponding 3D coordinates using the depth data from the Kinect v2.
  • Displaying these 3D coordinates on a point cloud image for visualization.

The Kinect v2 camera itself undergoes calibration using the Zhang Zhengyou method to ensure accurate correspondence between RGB and depth data.

Datasets

The paper mentions using the COCO dataset for training the YOLOv3 algorithm. No new datasets were introduced or benchmarked in this study.

Key Insights and Ideas

  • Synergy of YOLOv3 and Depth Sensing: The paper demonstrates the power of combining a state-of-the-art object detector (YOLOv3) with depth sensing (Kinect v2) for precise 3D localization of agricultural targets. This is more effective than relying on RGB alone or traditional methods.
  • Feature Point Definition for Varied Shapes: The authors propose a flexible feature point definition method that can adapt to different crop shapes, including irregular ones like bananas, by defining points at the center of bounding boxes or diagonally linked corners. This is crucial for consistent robotic grasping.
  • Robustness to Environmental Conditions: The YOLOv3 algorithm, as tested in this paper, shows high recognition accuracy even under challenging conditions like varying light, partial occlusion by leaves, and mutual occlusion between crops. This highlights its practical applicability in real-world farming scenarios.
  • Quantitative Error Control: The study establishes a clear evaluation criterion for relative error (within 5%) and demonstrates that their method consistently achieves this, with absolute errors below 0.5 cm on each axis. This level of precision is vital for robotic manipulation.

Key Results

  • High Accuracy: The proposed method achieves a relative error of less than 5% for crop feature point localization and an absolute error of less than 0.5 cm on each axis (X, Y, Z).
  • Efficient Detection: The YOLOv3 algorithm provides real-time object detection capabilities.
  • Robustness: The system performs well even with variations in lighting, occlusion, and multiple targets.
  • Comparison: The method is shown to be simpler and more accurate than other approaches like RGB image segmentation and LTLS coordinate transformation, particularly in terms of complexity.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that for irregularly shaped crops like bananas, the defined feature point might not be precisely at the geometric center, but it's deemed acceptable within the error tolerance for robotic grasping.
  • The effectiveness of the method is demonstrated for specific crop types and within the operational range of the Kinect v2 sensor.
  • Future Work:
    • The authors explicitly state that the next research direction is to develop the manipulation capabilities of the robot arm to automatically move to the 3D coordinates of the target crop and perform the capture. This involves integrating the developed vision system with robotic control.

Summary updated

Abstract

Based on computer vision technology, this paper proposes a method for identifying and locating crops in order to successfully capture crops in the process of automatic crop picking. This method innovatively combines the YOLOv3 algorithm under the DarkNet framework with the point cloud image coordinate matching method, and can achieve the goal of this paper very well. Firstly, RGB (RGB is the color representing the three channels of red, green and blue) images and depth images are obtained by using the Kinect v2 depth camera. Secondly, the YOLOv3 algorithm is used to identify the various types of target crops in the RGB images, and the feature points of the target crops are determined. Finally, the 3D coordinates of the feature points are displayed on the point cloud images. Compared with other methods, this method of crop identification has high accuracy and small positioning error, which lays a good foundation for the subsequent harvesting of crops using mechanical arms. In summary, the method used in this paper can be considered effective.
BibTeX
@inproceedings{cui2019extraction,
  author = {Jingwen Cui and Jianping Zhang and Guiling Sun and Bowen Zheng},
  title = {{Extraction and Research of Crop Feature Points Based on Computer Vision}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=D8CdF6z50J},
}

Loading related content...

Figure