Estimating Phenotypic Traits From UAV Based RGB Imagery

Javier Ribera, Fangning He, Yuhao Chen, Ayman F. Habib, Edward J. DelparXiv 2018 · OpenReview

TL;DR

This paper presents a novel approach for estimating plant phenotypic traits, specifically for sorghum, using low-cost Unmanned Aerial Vehicles (UAVs) equipped with RGB cameras. By developing new methods for image rectification, leaf segmentation, and plant localization, the authors enable more efficient and automated phenotyping, moving beyond traditional labor-intensive methods.

What Problem Does This Paper Address?

Accurately characterizing plant physical properties, a process known as phenotyping, is crucial for predicting crop performance, biomass, and understanding environmental influences. Traditional phenotyping methods are often manual, time-consuming, and can be destructive. While image-based phenotyping is gaining traction, most existing systems are designed for controlled indoor or greenhouse environments. This paper addresses the challenge of performing accurate phenotyping of crops grown in real-world outdoor agricultural fields using affordable UAVs and standard RGB cameras, which are prone to distortions and require robust processing pipelines.

Methods

The authors propose a multi-stage pipeline for UAV-based RGB imagery phenotyping:

  • Camera Calibration: They utilize the USGS Simultaneous Multi-frame Analytical Calibration (SMAC) distortion model to estimate the internal characteristics of the UAV's wide-angle RGB camera, accounting for both radial and decentering lens distortions.
  • Orthophoto Generation:
    • Structure from Motion (SfM): A three-step SfM approach is employed to recover image Exterior Orientation Parameters (EOPs) and generate a sparse image-based point cloud. This involves estimating relative orientation parameters (ROPs) using Scale-Invariant Feature Transform (SIFT) and a novel triangulation technique that leverages prior UAV flight trajectory information.
  • RGB Orthophoto Mosaic: A Digital Elevation Model (DEM) is interpolated from the sparse point cloud. This DEM, along with the refined image EOPs and camera calibration parameters, is used to generate a rectified orthophoto mosaic of the entire sorghum field.
  • Image-Based Phenotyping:
    • Leaf Counting: A color-based segmentation method is applied to the HSV color space of the images. By defining thresholds for hue, saturation, and value, sorghum leaves are segmented from soil and panicles. The number of segmented pixels is then linearly related to the leaf count, assuming leaves have similar areas.
  • Model-Based Leaf Segmentation: To estimate individual leaf traits like length and width, a shape-based approach is introduced. This method uses leaf slices (lines connecting pixels at opposite leaf edges) and their gradient angles to identify and segment individual leaves, even in cases of discontinuity due to leaf crossing.
  • Plant Location Model: A statistical model is developed to segment plants and determine their locations. This model uses the color-based leaf segmentation mask and a probabilistic approach to estimate the plant's position, defined as the pixel coordinates where the stalk intersects the ground. The model accounts for the structured nature of agricultural fields by introducing a prior distribution for plant positions, considering inter-row and intra-row spacing.

Datasets

The experimental dataset was acquired using a DJI Phantom 2 UAV equipped with a GoPro Hero 3+ Black Edition camera at the Agronomy Center for Research and Education (ACRE) at Purdue University. The UAV flew at an altitude of approximately 15 meters. Two sets of images were collected:

  • June 24, 2015: 489 images used for orthophoto mosaic generation.
  • July 15, 2015: 497 images used for leaf segmentation and analysis. Ground Control Points (GCPs) and check points were surveyed using GPS for accurate georeferencing and bundle adjustment.

Key Insights and Ideas

  • Bridging the Gap: The paper successfully demonstrates that affordable UAVs and standard RGB cameras can be effectively used for detailed plant phenotyping in outdoor agricultural settings, overcoming limitations of indoor systems.
  • Robust Image Rectification: The combination of camera calibration, SfM for EOP recovery, and DEM-based orthophoto generation provides a robust pipeline for creating distortion-free, spatially accurate mosaics essential for subsequent analysis.
  • Color-Based Leaf Segmentation Efficacy: The HSV color space segmentation proves effective in distinguishing green/yellow sorghum leaves from brown soil and panicles, providing a simple yet powerful method for initial leaf identification.
  • Shape-Based Leaf Segmentation for Trait Extraction: The model-based leaf segmentation, using leaf slices and gradient angles, offers a promising avenue for extracting detailed leaf morphology traits, which are correlated with biomass.
  • Statistical Plant Localization: The probabilistic model for plant location estimation, incorporating field structure and prior knowledge of plant alignment, offers a more sophisticated approach than simple clustering, potentially improving accuracy in structured agricultural environments.

Key Results

  • The developed methods enable the generation of rectified orthophoto mosaics from UAV-based RGB imagery.
  • A color-based method was demonstrated for segmenting sorghum leaves, with experimentally determined thresholds (T1=30, T2=79, T3=30, T4=163 in HSV).
  • A shape-based method was proposed for segmenting individual leaves, capable of handling discontinuities.
  • A statistical model was presented for estimating plant locations, which can be used to derive inter-row and intra-row spacing.
  • The orthophoto mosaic generated on June 24, 2015, achieved RMSEs of 0.04 m (X), 0.03 m (Y), and 0.05 m (Z).
  • The orthophoto mosaic generated on July 15, 2015, achieved RMSEs of 0.03 m (X), 0.03 m (Y), and 0.04 m (Z).

Limitations and Future Work

  • Leaf Area Assumption: The leaf counting method assumes that all leaves have approximately the same area, which might not hold true across all growth stages.
  • Leaf Edge Distinction: The model-based leaf segmentation requires clearly distinguishable leaf edges and may struggle with significant overlap, potentially leading to under-segmentation.
  • Computational Complexity: The plant location model's cost function evaluation can be computationally intensive, necessitating cropping the mosaic into regions for analysis.
  • Future Work: The authors plan to integrate additional sensors like position and orientation systems and LiDAR. Future research will also focus on using leaf angle information for plant center estimation and exploring the use of leaf segmentation for plant height determination.

Summary updated

Abstract

In many agricultural applications one wants to characterize physical properties of plants and use the measurements to predict, for example biomass and environmental influence. This process is known as phenotyping. Traditional collection of phenotypic information is labor-intensive and time-consuming. Use of imagery is becoming popular for phenotyping. In this paper, we present methods to estimate traits of sorghum plants from RBG cameras on board of an unmanned aerial vehicle (UAV). The position and orientation of the imagery together with the coordinates of sparse points along the area of interest are derived through a new triangulation method. A rectified orthophoto mosaic is then generated from the imagery. The number of leaves is estimated and a model-based method to analyze the leaf morphology for leaf segmentation is proposed. We present a statistical model to find the location of each individual sorghum plant.
BibTeX
@misc{ribera2018estimating,
  author = {Javier Ribera and Fangning He and Yuhao Chen and Ayman F. Habib and Edward J. Delp},
  title = {{Estimating Phenotypic Traits From UAV Based RGB Imagery}},
  year = {2018},
  eprint = {1807.00498},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1807.00498},
}
Figure