TL;DR
This paper presents a novel approach for estimating plant phenotypic traits, specifically for sorghum, using low-cost Unmanned Aerial Vehicles (UAVs) equipped with RGB cameras. By developing new methods for image rectification, leaf segmentation, and plant localization, the authors enable more efficient and automated phenotyping, moving beyond traditional labor-intensive methods.
What Problem Does This Paper Address?
Accurately characterizing plant physical properties, a process known as phenotyping, is crucial for predicting crop performance, biomass, and understanding environmental influences. Traditional phenotyping methods are often manual, time-consuming, and can be destructive. While image-based phenotyping is gaining traction, most existing systems are designed for controlled indoor or greenhouse environments. This paper addresses the challenge of performing accurate phenotyping of crops grown in real-world outdoor agricultural fields using affordable UAVs and standard RGB cameras, which are prone to distortions and require robust processing pipelines.
Methods
The authors propose a multi-stage pipeline for UAV-based RGB imagery phenotyping:
- Camera Calibration: They utilize the USGS Simultaneous Multi-frame Analytical Calibration (SMAC) distortion model to estimate the internal characteristics of the UAV's wide-angle RGB camera, accounting for both radial and decentering lens distortions.
- Orthophoto Generation:
- Structure from Motion (SfM): A three-step SfM approach is employed to recover image Exterior Orientation Parameters (EOPs) and generate a sparse image-based point cloud. This involves estimating relative orientation parameters (ROPs) using Scale-Invariant Feature Transform (SIFT) and a novel triangulation technique that leverages prior UAV flight trajectory information.
- RGB Orthophoto Mosaic: A Digital Elevation Model (DEM) is interpolated from the sparse point cloud. This DEM, along with the refined image EOPs and camera calibration parameters, is used to generate a rectified orthophoto mosaic of the entire sorghum field.
- Image-Based Phenotyping:
- Leaf Counting: A color-based segmentation method is applied to the HSV color space of the images. By defining thresholds for hue, saturation, and value, sorghum leaves are segmented from soil and panicles. The number of segmented pixels is then linearly related to the leaf count, assuming leaves have similar areas.
- Model-Based Leaf Segmentation: To estimate individual leaf traits like length and width, a shape-based approach is introduced. This method uses leaf slices (lines connecting pixels at opposite leaf edges) and their gradient angles to identify and segment individual leaves, even in cases of discontinuity due to leaf crossing.
- Plant Location Model: A statistical model is developed to segment plants and determine their locations. This model uses the color-based leaf segmentation mask and a probabilistic approach to estimate the plant's position, defined as the pixel coordinates where the stalk intersects the ground. The model accounts for the structured nature of agricultural fields by introducing a prior distribution for plant positions, considering inter-row and intra-row spacing.
Datasets
The experimental dataset was acquired using a DJI Phantom 2 UAV equipped with a GoPro Hero 3+ Black Edition camera at the Agronomy Center for Research and Education (ACRE) at Purdue University. The UAV flew at an altitude of approximately 15 meters. Two sets of images were collected:
- June 24, 2015: 489 images used for orthophoto mosaic generation.
- July 15, 2015: 497 images used for leaf segmentation and analysis. Ground Control Points (GCPs) and check points were surveyed using GPS for accurate georeferencing and bundle adjustment.
Key Insights and Ideas
- Bridging the Gap: The paper successfully demonstrates that affordable UAVs and standard RGB cameras can be effectively used for detailed plant phenotyping in outdoor agricultural settings, overcoming limitations of indoor systems.
- Robust Image Rectification: The combination of camera calibration, SfM for EOP recovery, and DEM-based orthophoto generation provides a robust pipeline for creating distortion-free, spatially accurate mosaics essential for subsequent analysis.
- Color-Based Leaf Segmentation Efficacy: The HSV color space segmentation proves effective in distinguishing green/yellow sorghum leaves from brown soil and panicles, providing a simple yet powerful method for initial leaf identification.
- Shape-Based Leaf Segmentation for Trait Extraction: The model-based leaf segmentation, using leaf slices and gradient angles, offers a promising avenue for extracting detailed leaf morphology traits, which are correlated with biomass.
- Statistical Plant Localization: The probabilistic model for plant location estimation, incorporating field structure and prior knowledge of plant alignment, offers a more sophisticated approach than simple clustering, potentially improving accuracy in structured agricultural environments.
Key Results
- The developed methods enable the generation of rectified orthophoto mosaics from UAV-based RGB imagery.
- A color-based method was demonstrated for segmenting sorghum leaves, with experimentally determined thresholds (T1=30, T2=79, T3=30, T4=163 in HSV).
- A shape-based method was proposed for segmenting individual leaves, capable of handling discontinuities.
- A statistical model was presented for estimating plant locations, which can be used to derive inter-row and intra-row spacing.
- The orthophoto mosaic generated on June 24, 2015, achieved RMSEs of 0.04 m (X), 0.03 m (Y), and 0.05 m (Z).
- The orthophoto mosaic generated on July 15, 2015, achieved RMSEs of 0.03 m (X), 0.03 m (Y), and 0.04 m (Z).
Limitations and Future Work
- Leaf Area Assumption: The leaf counting method assumes that all leaves have approximately the same area, which might not hold true across all growth stages.
- Leaf Edge Distinction: The model-based leaf segmentation requires clearly distinguishable leaf edges and may struggle with significant overlap, potentially leading to under-segmentation.
- Computational Complexity: The plant location model's cost function evaluation can be computationally intensive, necessitating cropping the mosaic into regions for analysis.
- Future Work: The authors plan to integrate additional sensors like position and orientation systems and LiDAR. Future research will also focus on using leaf angle information for plant center estimation and exploring the use of leaf segmentation for plant height determination.
Summary updated