High-throughput robotic plant phenotyping using 3D machine vision and deep neural networks

Lirong Xiang2022 · DOI

TL;DR

This dissertation presents a novel approach to high-throughput plant phenotyping using 3D machine vision and deep neural networks. By developing custom stereo imaging systems and sophisticated image processing pipelines, the research enables accurate, automated measurement of key plant traits in both controlled environments and challenging field conditions, paving the way for accelerated plant breeding.

What Problem Does This Paper Address?

Traditional plant phenotyping, the process of measuring plant characteristics, is a critical but often slow, labor-intensive, and error-prone bottleneck in plant breeding. This is especially true for complex traits like plant architecture, stem diameter, and leaf angle, which are crucial for understanding crop performance and developing improved varieties. Existing methods often rely on manual measurements or 2D imaging, which struggle with accuracy, occlusion, and the dynamic nature of plants in real-world field conditions. This work aims to overcome these limitations by developing automated, 3D vision-based systems for precise and efficient plant trait extraction.

Methods

The research employs a multi-faceted approach combining custom hardware development with advanced deep learning and computer vision techniques:

  • 3D Scanning System (Controlled Environments): A non-destructive system using a commodity depth camera (Kinect v2) was developed to capture sequential side-view images of sorghum plants. This system utilizes a vertical track linear actuator to acquire data at multiple heights.
  • 3D Skeletonization Algorithm: To analyze plant architecture and segment individual leaves from the 3D point cloud data, a novel skeletonization algorithm was created. This algorithm slices the point cloud and links Euclidean clusters, enabling the separation of overlapping tillers and individual leaves.
  • PhenoStereo Module (Field Conditions): A custom-built, robust stereo imaging module was designed for high-throughput field phenotyping. This module integrates strobe lighting to overcome variable outdoor lighting and wind conditions, enabling high-speed stereoscopic image capture.
  • Mask R-CNN for Stem Segmentation: A Mask Region Convolutional Neural Network (Mask R-CNN) was employed to detect and segment individual sorghum stems from RGB images. This allows for precise localization of stems for diameter measurement.
  • Semi-Global Block Matching (SGBM): This algorithm was used for stereo matching to generate disparity maps, which are crucial for reconstructing 3D point clouds from the stereo image pairs captured by PhenoStereo.
  • Cylindricity Feature for Filtering: A novel local feature descriptor, "Cylindricity," was defined and used to discriminate stem points from non-stem points (like leaf collars) in the point cloud, improving the accuracy of stem segmentation.
  • AngleNet Pipeline (Maize Leaf Angle): A unique image processing pipeline was developed for maize leaf angle detection. This pipeline uses a CNN to detect leaf collar regions and keypoints, then reprojects these into 3D space for angle calculation. It leverages a modified CenterNet architecture for anchor-free object detection.
  • 3D Reconstruction and Triangulation: Standard techniques like camera calibration, rectification, stereo matching (using 3D-MST), and triangulation are used to generate dense 3D point clouds of plant canopies.

Datasets

  • Sorghum Plants (Controlled Environments): Data was collected from 32 sorghum plants across four genotypes, sampled at multiple developmental timepoints (W3, W4, W5). Ground truth data was collected manually for plant height, leaf angle, stem diameter, leaf area, and biomass.
  • Maize Plants (Field Conditions):
    • PS4-Boone Dataset: Collected at Iowa State University's Agronomy Farm, featuring 10 maize plants from the B73 and Mo17 inbred lines, with additional randomly selected lines.
  • PS3-Ames Dataset: Collected at Iowa State University's Curtiss Farm, featuring 6 maize plants.
  • Annotation: For AngleNet, 620 images were manually annotated with bounding boxes around leaf collars, and 240 images were annotated with leaf angle keypoints. Datasets were split into training, validation, and test sets at a 7:2:1 ratio. Image augmentation techniques were applied.

Key Insights and Ideas

  • Stereo Vision as a Robust Field Solution: The research strongly advocates for customized stereo vision systems, like PhenoStereo, as a highly desirable and feasible method for field-based plant phenotyping, capable of overcoming challenges like variable lighting and wind.
  • Deep Learning for Keypoint Detection: The AngleNet pipeline demonstrates the power of CNNs for keypoint detection (leaf collar, midrib, stem) as a robust way to estimate leaf angles, even with varying leaf orientations and occlusions. This approach bypasses complex segmentation steps.
  • Cylindricity for Stem Discrimination: The introduction of the "Cylindricity" feature provides an effective geometric descriptor to distinguish plant stems from other plant parts, improving the accuracy of stem segmentation and diameter estimation.
  • 3D Skeletonization for Plant Architecture: The graph-based 3D skeletonization algorithm offers a powerful way to analyze plant architecture and extract multiple phenotypic parameters from point cloud data, even in the presence of overlapping leaves and tillers.
  • Stem Volume as a Biomass Predictor: Stem volume, derived from 3D measurements, emerged as a strong predictor of shoot fresh and dry weight, highlighting its importance in biomass estimation.

Key Results

  • Sorghum Plant Height & Leaf Area: System-derived traits showed high correlations with ground truth (r = 0.81 for plant height, r = 0.94 for leaf area).
  • Sorghum Stem Diameter: Achieved a strong correlation (r = 0.97) with ground truth, with an RMSE of 1.39 mm, outperforming previous methods.
  • Sorghum Biomass Prediction: Total leaf area and stem volume showed good potential for predicting wet and dry biomass (r = 0.82 ~ 0.98).
  • Maize Leaf Angle: AngleNet-derived leaf angles were highly correlated with in-field measurements (r > 0.87, RMSE ≤ 4°).
  • Maize Node Height: Highly correlated with manual measurements (r > 0.99).
  • Detection Performance (AngleNet): Achieved an AP of 0.908 at IoU(0.5) for leaf collar detection. Keypoint detection showed deviations less than 3% of the leaf angle skeleton length.
  • Computational Efficiency: The automated stem diameter estimation pipeline was significantly faster than manual measurements.

Limitations and Future Work

  • Leaf Angle Limitations: AngleNet struggles with wavy-shaped leaves, where reprojecting 2D lines to 3D can be unreliable. The accuracy of leaf angle estimation can also be affected by errors in ground-truth collection and the inherent difficulty in reconstructing accurate 3D models from stereo images.
  • Stem Diameter Limitations: The approach showed lower reliability for stems with diameters less than 20 mm, possibly due to the difficulty in reconstructing thin structures and the elliptical nature of small stems.
  • PhenoStereo Improvements: Future work could involve adding polarizing filters to reduce glare from reflective plant surfaces. Using multiple camera sets with varying horizontal viewing angles could improve coverage and mitigate occlusions. Optimizing the strobe lighting pattern for uniform distribution across the entire image is also suggested for traits like leaf area.
  • Advanced Stereo Matching: Employing more advanced stereo matching algorithms (e.g., CNN-based methods like MC-CNN) could further improve disparity map quality and surface reconstruction.
  • Broader Applications: The developed systems and algorithms could be extended to characterize other organ-level traits (e.g., leaf angle and panicle/tassel size in maize) and applied to other plant species and even forestry studies.
  • AI Integration: Future research should focus on developing multi-view 3D reconstruction systems combined with AI for object detection and segmentation to better handle challenges like leaf overlapping, occlusions, and homogenous textures.

Summary updated

Abstract

The ability to correlate morphological traits of plants with their genotypes plays an important role in plant phenomics research. However, traditional plant phenotyping is time-consuming, labor-intensive, and prone to human errors. This dissertation documents my innovative research in high-throughput robotic plant phenotyping for sorghum and maize plants using 3D machine vision and convolutional neural networks. Sorghum is an important grain crop and a promising feedstock for biofuel production due to its excellent drought tolerance and water use efficiency. The 3D surface model of a plant can potentially provide an efficient and accurate way to digitize plant architecture and accelerate sorghum plant breeding programs. A non-destructive 3D scanning system using a commodity depth camera was developed to take side-view images of plants at multiple growth stages. A 3D skeletonization algorithm was developed to analyze the plant architecture and segment individual leaves. Multiple phenotypic parameters were obtained from the skeleton and the reconstructed point cloud including plant height, stem diameter, leaf angle, and leaf surface area. These image-derived features were highly correlated with the ground truth. Additionally, the results showed that stem volume was a promising predictor of shoot fresh weight and shoot dry weight. To address the challenges of in-field imaging for plant phenotyping caused by variable outdoor lighting, wind conditions, and occlusions of plants. A customized stereo module, namely PhenoStereo, was developed for acquiring high-quality image data under field conditions. PhenoStereo was used to acquire a set of sorghum plant images and an automated point cloud data processing pipeline was also developed to automatically extract the stems and then quantify their diameters via an optimized 3D modeling process. The pipeline employed a Mask Region Convolutional Neural Network for detecting stalk contours and a Semi-Global Block Matching stereo matching algorithm for generating disparity maps. The system-derived stem diameters were highly correlated with the ground truth. Additionally, PhenoStereo was used to quantify the leaf angle of maize plants under field conditions. Multiple tiers of PhenoStereo camera were mounted on PhenoBot 3.0, a robotic vehicle designed to traverse between pairs of agronomically spaced rows of crops, to capture side-view images of maize plants in the field. An automated image processing pipeline (AngleNet) was developed to detect each leaf angle as a triplet of keypoints in two-dimensional images and extract quantitative data from reconstructed 3D models. AngleNet-derived leaf angles and their associated internode heights were highly correlated with manually collected ground-truth measurements. The dissertation investigates and develops automated computer-vision-based robotic systems for plant phenotyping under controlled environments and in field conditions. In particular, a stereo module was customized and utilized to acquire high-quality image data for in-field plant phenotyping. With high-fidelity reconstructed 3D models and robust image processing algorithms, a series of plant-level and organ-level phenotypic traits of sorghum and maize plants were accurately extracted. The results demonstrated that with proper customization stereo vision can be a highly desirable sensing method for field-based plant phenotyping using high-fidelity 3D models reconstructed from stereoscopic images. The proposed approaches in this dissertation provide efficient alternatives to traditional phenotyping that could potentially accelerate breeding programs for improved plant architecture.
BibTeX
@misc{xiang2022high,
  author = {Lirong Xiang},
  title = {{High-throughput robotic plant phenotyping using 3D machine vision and deep neural networks}},
  year = {2022},
  doi = {10.31274/td-20240329-387},
}
Figure