In-Field Phenotyping Based on Crop Leaf and Plant Instance Segmentation

Jan Weyler, Federico Magistri, Peter Seitz, Jens Behley, Cyrill StachnissWACV 2022

TL;DR

This paper introduces a novel vision-based approach for simultaneously segmenting individual crop leaves and entire plants in real-world agricultural fields. By accurately identifying and associating each leaf with its plant, the system enables precise, automated phenotyping that can significantly aid plant scientists and breeders in understanding crop growth and performance.

What Problem Does This Paper Address?

Understanding a plant's phenotype – its observable characteristics – is crucial for plant scientists and breeders to develop improved crop varieties. Traditionally, this involves manual assessment in the field, which is time-consuming, labor-intensive, and prone to subjectivity. Vision-based systems offer a promising alternative for objective, high-resolution, and large-scale assessment. A key aspect of phenotyping is tracking leaf development, as leaf count, size, and coverage are strong indicators of growth stage and yield potential. However, accurately segmenting individual leaves and, critically, associating them with their specific parent plant in complex, uncontrolled field environments (with multiple plants, varying lighting, and shadows) has been a significant challenge. Existing methods often focus on either semantic segmentation (classifying pixels) or instance segmentation (detecting individual objects) but rarely achieve both simultaneously for both leaves and plants, hindering the extraction of per-plant phenotypic traits.

Methods

The core of the proposed approach is a single-shot convolutional neural network (CNN) designed for simultaneous instance segmentation of crop leaves and plants. The network takes RGB images as input and, through an encoder-decoder architecture (based on ERFNet), generates several key outputs:

  • Offset Maps ($\Delta_L$, $\Delta_P$): These maps guide pixels towards the center of their associated leaf or plant instance, respectively.
  • Spatial Embeddings: By combining the offset maps with pixel coordinates, the network creates representations where pixels are translated towards their instance centers.
  • Instance Covariance Matrices ($\Sigma_L$, $\Sigma_P$): This is a novel contribution. Instead of relying on simple diagonal covariance matrices (which assume axis-aligned clustering regions), the network predicts full covariance matrices. This allows the clustering regions to adapt to the specific shape and orientation of individual leaves and plants, leading to more accurate segmentation.
  • Score Maps ($S_L$, $S_P$): These maps predict the likelihood of a pixel belonging to a leaf or plant instance, used to recover instance centers during post-processing.

The post-processing step involves a bottom-up clustering approach. First, crop leaves are clustered based on their spatial embeddings and predicted covariance matrices. Then, these leaf clusters are used to identify and cluster individual crop plants, effectively defining a plant as the union of its constituent leaves. This ensures that each segmented leaf is associated with a single plant.

Datasets

The primary dataset used for training and evaluation consists of RGB images of sugar beet fields. This dataset comprises 1316 images, each with a resolution of 1024 px × 512 px and a ground sampling distance of 1 mm. The images were captured by a UAV in real-world, uncontrolled field conditions, presenting challenges like shadows and variable illumination. The authors also evaluated their method on the CVPPP Leaf Segmentation Challenge (LSC) dataset, a popular benchmark for leaf segmentation.

Key Insights and Ideas

  • Simultaneous Segmentation is Key: The paper highlights that performing leaf and plant instance segmentation simultaneously is crucial for extracting per-plant phenotypic traits like leaf count.
  • Full Covariance Matrices for Flexible Clustering: The use of full covariance matrices to define clustering regions is a significant innovation. It allows the segmentation to adapt to the varying shapes and orientations of leaves and plants in the field, outperforming methods that rely on simpler, axis-aligned covariance matrices.
  • Bottom-Up Approach for Natural Grouping: The bottom-up strategy, where leaves are first segmented and then grouped into plants, naturally models the hierarchical structure of plants and their leaves.
  • Real-World Field Performance: The approach is validated on challenging, real-world agricultural field data, demonstrating its robustness beyond controlled laboratory settings.

Key Results

The proposed method demonstrates superior performance compared to state-of-the-art approaches like Mask R-CNN and Harmonic Embeddings.

  • Leaf Count Accuracy: The method achieves higher accuracy in terms of leaf count per crop plant, as measured by the Absolute Difference in Count (|DiC|) and Percentage Agreement (Pa) metrics.
  • Segmentation Quality: The predicted masks for crop leaves and plants show improved accuracy, with the Symmetric Best Dice (SBD) for leaves and Foreground-Background Dice (FBD) for plants outperforming baselines. For instance, the SBD for crop leaves is significantly higher than Mask R-CNN.
  • Qualitative Improvements: Visual results show that the predicted masks are consistent, with plants being modeled as the union of their leaves. The approach is also less prone to confusing crop plants/leaves with weeds.
  • Ablation Studies: Experiments confirm that using full covariance matrices and offset maps significantly improves segmentation performance, demonstrating the effectiveness of these novel components.

Limitations and Future Work

The paper acknowledges that while their approach is competitive on the CVPPP LSC benchmark, that benchmark assumes a single plant per image, a simplification not present in their primary dataset. Their method's ability to handle multiple plants in a single image makes it more broadly applicable. Future work could explore extending this approach to more complex plant architectures or integrating it with other sensor modalities for even richer phenotyping data. The authors also suggest further investigation into optimizing the clustering process and exploring more advanced network architectures.

Summary updated

BibTeX
@inproceedings{weyler2022field,
  author = {Jan Weyler and Federico Magistri and Peter Seitz and Jens Behley and Cyrill Stachniss},
  title = {{In-Field Phenotyping Based on Crop Leaf and Plant Instance Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure