TL;DR
This paper presents an autonomous fruit and vegetable harvesting system that utilizes a novel, low-cost gripper and a Mask R-CNN-based vision system for precise peduncle detection and cutting. The system aims to overcome the challenges of unstructured agricultural environments, offering a more efficient and less damaging harvesting solution.
What Problem Does This Paper Address?
Manual harvesting of fruits and vegetables is labor-intensive and time-consuming. While autonomous harvesting robots offer potential benefits like reduced labor, improved quality, and better environmental control, their widespread adoption is hindered by several key challenges. These include the difficulty of reliably detecting and precisely cutting the peduncle (the stalk connecting the fruit to the plant) in complex, unstructured environments with varying lighting and occlusions. Furthermore, existing end-effectors that combine grasping and cutting functions are often expensive, bulky, or require separate actuation systems, increasing complexity and cost. This paper tackles these issues by proposing a system that is robust, cost-effective, and capable of harvesting a variety of crops by accurately identifying and severing the peduncle.
Methods
The proposed system comprises two main modules: a detection module and a picking module.
- Detection Module: This module leverages the state-of-the-art Mask Region-based Convolutional Neural Network (Mask R-CNN) for object detection and instance segmentation. The authors fine-tune Mask R-CNN to identify fruits and their bounding boxes. A geometric approach is then applied to the bounding box information and depth data (from a Kinect sensor) to determine the precise cutting point on the peduncle. This involves defining a Region of Interest (ROI) above the fruit and using geometric calculations to pinpoint the cutting location.
- Picking Module: This module features a novel, low-cost gripper designed to simultaneously clamp and cut the peduncle. Inspired by human dexterity, the gripper consists of three main parts: a top plate with a cutting blade, a middle plate for pressing, and a bottom plate for holding. Crucially, the clamping and cutting mechanisms share a single power-source drive, significantly reducing complexity and cost. The paper includes a detailed force analysis of the gripper, outlining the moments and forces involved in clamping and cutting.
- Control: For motion planning, the system employs a learning-based kinematic control policy using Convolutional Neural Networks (CNNs). This approach maps target end-effector positions to joint actions, bypassing the need for precise manipulator kinematic and dynamic models, which are often difficult to obtain for custom robots. This learned policy also considers self-collision avoidance.
Datasets
The paper describes using a dataset of 60 images per crop type for training and noise introduction. These images were collected from Google and labeled with bounding box information for each crop. For testing, 15 images per crop type were used, taken from various viewpoints. The authors also mention using both real and plastic crops for experiments, including specific counts of various fruits. No new benchmark datasets were introduced.
Key Insights and Ideas
- Low-Cost, Integrated Gripper Design: The simultaneous clamping and cutting mechanism sharing a single drive is a key innovation. This significantly simplifies the end-effector, reducing cost and complexity, which are major hurdles for agricultural robotics.
- Geometric Cutting Point Detection with Mask R-CNN: Combining the powerful segmentation capabilities of Mask R-CNN with a geometric approach for peduncle cutting point detection provides a robust method for identifying the precise location to cut, even in cluttered environments.
- Learning-Based Kinematic Control: The use of CNNs to learn an inverse kinematics function that maps target positions to joint actions is a significant advantage. It allows the system to adapt to different robots and configurations without requiring explicit modeling, making it more versatile.
- Focus on Peduncle Cutting: The paper emphasizes the importance of cutting at the peduncle for maximizing crop storage life and market value, a critical factor in commercial harvesting.
Key Results
- Detection Success Rates:
- Plastic crops: 93% detection success, 89% cutting-point detection success.
- Real crops: 87% detection success, 71% cutting-point detection success.
- Harvesting Success Rates (Overall):
- Plastic crops: Attachment success rate of 81%, detachment success rate of 70%.
- Real crops: Attachment success rate of 61%, detachment success rate of 55%.
- Failure Analysis: The most frequent failure modes were identified as: (2) cutting point not detected (especially for irregularly shaped crops) and (5) crop dropped due to insufficient clamping. For real crops, issues with peduncle stiffness and gripper thickness also contributed to detachment failures.
- Modified Scenario (Leaf Removal): In a scenario where leaves occluded fruits, the robot successfully removed leaves to access the cutting point, though this operation had a lower success rate (4 attempts, 2 failures).
Limitations and Future Work
- Limitations:
- The current system is not suitable for field testing and was conducted in a laboratory environment.
- The gripper's thickness and stiffness were identified as limitations for cutting thicker peduncles and clamping certain crops.
- The system struggled with irregularly shaped crops and cases where peduncles were short.
- The current motion planning does not explicitly avoid obstacles like unripened crops or stems.
- Cases where the crop is parallel to the camera or occluded by its own flesh were not fully addressed.
- Future Work:
- Devise new approaches for detecting cutting poses for irregularly shaped crops.
- Improve the gripper design using metal materials and a better cutting blade to enhance stiffness and clamping ability for a wider range of crops.
- Develop vision-guided motion planning using deep reinforcement learning to reach desired fruits and avoid obstacles.
- Integrate a "next-best-view" approach to determine optimal camera angles for occluded crops.
- Test the system in the field with more crop types and consider autonomous reconstruction of unknown scenes.
- Improve detection of occluded crops and leaves, and refine robot grasping poses.
Summary updated