TL;DR
This paper introduces a low-cost, machine vision system for real-time yield monitoring of shallots during harvest. The system uses an RGB camera and computer vision algorithms to count shallots and classify them by size, aiming to provide valuable data for precision agriculture practices in specialty crops.
What Problem Does This Paper Address?
Precision agriculture (PA) techniques are crucial for optimizing resource management and improving crop yields. While PA has seen significant advancements in row crops like corn and soybeans, specialty crops, such as fruits and vegetables, often lag behind. This is partly due to the diversity of harvesting methods and smaller market sizes. Current yield estimation for specialty crops is often manual, labor-intensive, and prone to inaccuracies. The paper addresses the gap in developing automated, real-time yield monitoring systems for vegetable crops, specifically shallots, to provide detailed harvest information and enable better quality assessment and management.
Methods
The core of the system is a machine vision pipeline designed to operate in-situ on a commercial shallot harvester. Key components include:
- Hardware: An RGB camera (ELP 1080P USB Camera Box) for image capture and a GPS sensor (Garmin 19x HVS) for geo-tagging. Processing is done on a compact computer (MINIX NEO-Z83-4-PRO-VESA). The entire system is housed in a rugged, watertight case.
- Data Collection: Real-time video data is collected at 640x480 resolution and 120 frames/s under natural sunlight conditions.
- Image Processing Pipeline:
- Preprocessing: Images are first blurred using median and Gaussian filters to reduce noise.
- Color Segmentation: The image is converted to the HSV color space, and color thresholding is used to identify onion regions and specular reflection areas.
- Morphological Operations: Opening and closing operations refine the segmented regions.
- Watershed Segmentation: A marker-controlled watershed transform is applied to separate adjacent or overlapping onion regions. This method interprets the image gradient as a topographical surface, using catchment basins to segment regions.
- Size Classification: After segmentation, properties like size and contour length are assessed. Shallots are then classified into three size categories (small, medium, large) based on pre-defined diameter thresholds derived from manual measurements and correlation with weight.
- Size Calibration: A reference object (a tennis ball) is used to establish a pixel-to-metric ratio for accurate size estimation.
Datasets
- Data Collection: Images were collected on a sunny day during the shallot harvesting season in Napierville, Quebec.
- Evaluation Dataset: A subset of 246 images (20% of the total collected) was randomly selected for evaluation. Within this subset, 1,782 onions were manually identified and used as ground truth for detection performance.
- Size Calibration: 35 onions were manually measured (diameter and weight) to establish size categories. A standard 300mm ruler was used for initial calibration tests, and a tennis ball was used for the field trial.
Key Insights and Ideas
- Feasibility of Real-time Vegetable Yield Monitoring: The study demonstrates that a relatively simple and inexpensive machine vision system can be integrated into existing harvesting equipment to provide real-time yield data for specialty crops like shallots.
- Watershed Segmentation for Overlapping Objects: The use of watershed segmentation proved effective in separating individual shallots, even when they were clustered or overlapping, which is a common challenge in harvesting.
- Size Classification Challenges: While the system performed well for large shallots, accuracy dropped for small and medium sizes. This highlights the difficulty in precisely defining size boundaries and the impact of factors like occlusion and lighting.
- Low-Cost System Potential: The system's low cost ($1212 CAD) makes it an attractive option for producers who might not have access to more expensive traditional agricultural machinery.
- Importance of Robustness: The research underscores the need for systems that are robust to variations in lighting, occlusion, and other environmental factors common in field operations.
Key Results
- Onion Detection: The system achieved a precision of 76.0% and a recall of 72.7% for detecting shallots. Overall accuracy was 59.2%.
- Size Classification:
- Large class: 73.3% accuracy.
- Small class: 58.6% accuracy.
- Medium class: 44.4% accuracy.
- Size Calibration: The size calibration method using a reference object showed a high correlation (R² = 0.94) with true size measurements, with a standard deviation of 2.33 mm.
- Error Analysis: The system performed better when onion counts per image were low. As the count increased beyond nine, detection accuracy decreased, likely due to segmentation issues with cluttered images.
- Statistical Analysis (KS Test): The null hypothesis (that predicted and true diameters are from the same distribution) was not rejected for the large class (p > 0.05), indicating acceptable prediction for this category. However, it was rejected for the small and medium classes (p < 0.05).
Limitations and Future Work
- Size Classification Accuracy: The system's performance in classifying small and medium shallots was limited, with accuracy dropping significantly. This is attributed to issues with segmenting bulbs on the image border, overlapping onions, occlusion by stems, and shadowy regions.
- Lighting and Occlusion: Inconsistent lighting conditions and the occasional occlusion of vegetables were identified as the main factors hindering performance.
- Calibration Robustness: The current calibration method relies on pre-set thresholds for a reference object, which can be affected by varying lighting conditions.
- Future Improvements:
- Adaptive Calibration: Developing a field-adjustable threshold setting for color segmentation would improve robustness to lighting.
- Advanced Segmentation: Exploring machine learning techniques like fully-connected Convolutional Neural Networks (CNNs) for semantic segmentation could improve feature learning and accommodate variability.
- Unsupervised Learning: Investigating K-Means clustering or Decision Trees for segmentation is suggested as an alternative.
- Tracking: Implementing visual tracking algorithms (e.g., Kalman Filter) to analyze sequential frames could improve overall detection rates and reduce false positives.
- Hardware Reliability: Addressing power connection issues and using a more robust, weatherproof camera would enhance system reliability in harsh agricultural environments.
- Non-linear Size Classification: Considering non-linear methods or modeling vegetables as ellipses could improve size class separation.
Summary updated