TL;DR
This dissertation presents a novel approach to high-throughput plant phenotyping using 3D machine vision and deep neural networks. By developing custom stereo imaging systems and sophisticated image processing pipelines, the research enables accurate, automated measurement of key plant traits in both controlled environments and challenging field conditions, paving the way for accelerated plant breeding.
What Problem Does This Paper Address?
Traditional plant phenotyping, the process of measuring plant characteristics, is a critical but often slow, labor-intensive, and error-prone bottleneck in plant breeding. This is especially true for complex traits like plant architecture, stem diameter, and leaf angle, which are crucial for understanding crop performance and developing improved varieties. Existing methods often rely on manual measurements or 2D imaging, which struggle with accuracy, occlusion, and the dynamic nature of plants in real-world field conditions. This work aims to overcome these limitations by developing automated, 3D vision-based systems for precise and efficient plant trait extraction.
Methods
The research employs a multi-faceted approach combining custom hardware development with advanced deep learning and computer vision techniques:
- 3D Scanning System (Controlled Environments): A non-destructive system using a commodity depth camera (Kinect v2) was developed to capture sequential side-view images of sorghum plants. This system utilizes a vertical track linear actuator to acquire data at multiple heights.
- 3D Skeletonization Algorithm: To analyze plant architecture and segment individual leaves from the 3D point cloud data, a novel skeletonization algorithm was created. This algorithm slices the point cloud and links Euclidean clusters, enabling the separation of overlapping tillers and individual leaves.
- PhenoStereo Module (Field Conditions): A custom-built, robust stereo imaging module was designed for high-throughput field phenotyping. This module integrates strobe lighting to overcome variable outdoor lighting and wind conditions, enabling high-speed stereoscopic image capture.
- Mask R-CNN for Stem Segmentation: A Mask Region Convolutional Neural Network (Mask R-CNN) was employed to detect and segment individual sorghum stems from RGB images. This allows for precise localization of stems for diameter measurement.
- Semi-Global Block Matching (SGBM): This algorithm was used for stereo matching to generate disparity maps, which are crucial for reconstructing 3D point clouds from the stereo image pairs captured by PhenoStereo.
- Cylindricity Feature for Filtering: A novel local feature descriptor, "Cylindricity," was defined and used to discriminate stem points from non-stem points (like leaf collars) in the point cloud, improving the accuracy of stem segmentation.
- AngleNet Pipeline (Maize Leaf Angle): A unique image processing pipeline was developed for maize leaf angle detection. This pipeline uses a CNN to detect leaf collar regions and keypoints, then reprojects these into 3D space for angle calculation. It leverages a modified CenterNet architecture for anchor-free object detection.
- 3D Reconstruction and Triangulation: Standard techniques like camera calibration, rectification, stereo matching (using 3D-MST), and triangulation are used to generate dense 3D point clouds of plant canopies.
Datasets
- Sorghum Plants (Controlled Environments): Data was collected from 32 sorghum plants across four genotypes, sampled at multiple developmental timepoints (W3, W4, W5). Ground truth data was collected manually for plant height, leaf angle, stem diameter, leaf area, and biomass.
- Maize Plants (Field Conditions):
- PS4-Boone Dataset: Collected at Iowa State University's Agronomy Farm, featuring 10 maize plants from the B73 and Mo17 inbred lines, with additional randomly selected lines.
- PS3-Ames Dataset: Collected at Iowa State University's Curtiss Farm, featuring 6 maize plants.
- Annotation: For AngleNet, 620 images were manually annotated with bounding boxes around leaf collars, and 240 images were annotated with leaf angle keypoints. Datasets were split into training, validation, and test sets at a 7:2:1 ratio. Image augmentation techniques were applied.
Key Insights and Ideas
- Stereo Vision as a Robust Field Solution: The research strongly advocates for customized stereo vision systems, like PhenoStereo, as a highly desirable and feasible method for field-based plant phenotyping, capable of overcoming challenges like variable lighting and wind.
- Deep Learning for Keypoint Detection: The AngleNet pipeline demonstrates the power of CNNs for keypoint detection (leaf collar, midrib, stem) as a robust way to estimate leaf angles, even with varying leaf orientations and occlusions. This approach bypasses complex segmentation steps.
- Cylindricity for Stem Discrimination: The introduction of the "Cylindricity" feature provides an effective geometric descriptor to distinguish plant stems from other plant parts, improving the accuracy of stem segmentation and diameter estimation.
- 3D Skeletonization for Plant Architecture: The graph-based 3D skeletonization algorithm offers a powerful way to analyze plant architecture and extract multiple phenotypic parameters from point cloud data, even in the presence of overlapping leaves and tillers.
- Stem Volume as a Biomass Predictor: Stem volume, derived from 3D measurements, emerged as a strong predictor of shoot fresh and dry weight, highlighting its importance in biomass estimation.
Key Results
- Sorghum Plant Height & Leaf Area: System-derived traits showed high correlations with ground truth (r = 0.81 for plant height, r = 0.94 for leaf area).
- Sorghum Stem Diameter: Achieved a strong correlation (r = 0.97) with ground truth, with an RMSE of 1.39 mm, outperforming previous methods.
- Sorghum Biomass Prediction: Total leaf area and stem volume showed good potential for predicting wet and dry biomass (r = 0.82 ~ 0.98).
- Maize Leaf Angle: AngleNet-derived leaf angles were highly correlated with in-field measurements (r > 0.87, RMSE ≤ 4°).
- Maize Node Height: Highly correlated with manual measurements (r > 0.99).
- Detection Performance (AngleNet): Achieved an AP of 0.908 at IoU(0.5) for leaf collar detection. Keypoint detection showed deviations less than 3% of the leaf angle skeleton length.
- Computational Efficiency: The automated stem diameter estimation pipeline was significantly faster than manual measurements.
Limitations and Future Work
- Leaf Angle Limitations: AngleNet struggles with wavy-shaped leaves, where reprojecting 2D lines to 3D can be unreliable. The accuracy of leaf angle estimation can also be affected by errors in ground-truth collection and the inherent difficulty in reconstructing accurate 3D models from stereo images.
- Stem Diameter Limitations: The approach showed lower reliability for stems with diameters less than 20 mm, possibly due to the difficulty in reconstructing thin structures and the elliptical nature of small stems.
- PhenoStereo Improvements: Future work could involve adding polarizing filters to reduce glare from reflective plant surfaces. Using multiple camera sets with varying horizontal viewing angles could improve coverage and mitigate occlusions. Optimizing the strobe lighting pattern for uniform distribution across the entire image is also suggested for traits like leaf area.
- Advanced Stereo Matching: Employing more advanced stereo matching algorithms (e.g., CNN-based methods like MC-CNN) could further improve disparity map quality and surface reconstruction.
- Broader Applications: The developed systems and algorithms could be extended to characterize other organ-level traits (e.g., leaf angle and panicle/tassel size in maize) and applied to other plant species and even forestry studies.
- AI Integration: Future research should focus on developing multi-view 3D reconstruction systems combined with AI for object detection and segmentation to better handle challenges like leaf overlapping, occlusions, and homogenous textures.
Summary updated