Cotton plant part 3D segmentation and architectural trait extraction using point voxel convolutional neural networks

Farah Saeed, Shangpeng Sun, Javier Rodriguez-Sanchez, John L. Snider, Tianming Liu, Changying LiPlant 2023 · DOI

TL;DR

Researchers have developed a high-efficiency 3D deep learning workflow to "see" and measure the complex architecture of cotton plants. By leveraging Point Voxel Convolutional Neural Networks (PVCNN), the system segments stems, branches, and bolls from LiDAR point clouds with 96% accuracy, enabling the automated extraction of seven critical growth traits in under a second.

What Problem Does This Paper Address?

In plant breeding, "architecture" is everything—it dictates how a plant intercepts light, resists disease, and ultimately produces yield. However, measuring traits like branch angles or the exact number of cotton bolls is traditionally a manual, error-prone "nightmare" for researchers.

While 2D computer vision has tried to automate this, it fails due to occlusion (leaves and branches hiding each other) and depth ambiguity. 3D LiDAR scans solve the occlusion problem, but they introduce a new challenge: stems and branches look nearly identical (both are tubular structures), making it difficult for standard algorithms to tell them apart. Existing 3D deep learning models are often either too slow for high-throughput use or lack the local detail needed to distinguish these similar parts.

Methods

The authors introduced a multi-stage pipeline:

  1. PlantCloud Annotation Tool: To train their models, they first built a custom C++/OpenGL tool designed to handle high-resolution point clouds (millions of points) with much lower memory overhead than existing web-based tools.
  2. PVCNN (The "Secret Sauce"): Unlike standard networks that use only points or only voxels, the Point Voxel CNN uses a hybrid approach. It uses a point-based branch to capture global features (the overall "shape" of the plant) and a voxel-based branch to perform efficient local convolutions. This allows the model to recognize fine-grained patterns (like the attachment point of a branch) without the massive memory cost of traditional 3D grids.
  3. Geometric Post-processing: Deep learning isn't perfect; it sometimes produces "label noise." The authors developed algorithms to enforce botanical logic—for example, correcting small segments of a main stem that were misclassified as branches based on their spatial orientation.
  4. Trait Extraction: They used DBSCAN (a density-based clustering algorithm) to count individual bolls and Principal Component Analysis (PCA) to calculate the precise inclination angles of branches.

Datasets

  • Primary Dataset: 30 individual cotton plants scanned using a FARO terrestrial LiDAR.
  • Diversity: The scans included plants from both single-plant plots (wider structure) and regular field plots (denser, more vertical).
  • Annotation: Over 450,000 points per plant were manually labeled into three classes: main stem, branches, and bolls.
  • Pre-training: The models were initially pre-trained on the ShapeNet dataset before being fine-tuned on the cotton data.

Key Insights and Ideas

  • Hybrid Representations Win: The study proves that combining point and voxel data is superior for plant phenotyping. Point-based models (like Pointnet++) are slow because they struggle with memory locality, while PVCNN achieved a 3.5x speedup over Pointnet++ while maintaining higher accuracy.
  • Geometry over Color: Interestingly, the model achieved these results using only XYZ coordinates, ignoring RGB color data. This makes the system robust to changing light conditions in the field or greenhouse.
  • The "Tube" Challenge: The most significant hurdle was distinguishing the main stem from branches. The authors found that global information is essential here; a local "slice" of a stem looks just like a branch, but the global context of the plant's vertical axis allows the PVCNN to differentiate them.

Key Results

  • Segmentation Performance: The PVCNN achieved a Mean Intersection over Union (mIoU) of 89.12% and an overall accuracy of 96.19%.
  • Inference Speed: The model processed a plant point cloud in just 0.88 seconds, making it viable for real-time applications.
  • Trait Accuracy:
    • Boll Counting: Achieved a near-perfect correlation ($R^2 = 0.995$) with manual counts.
  • Main Stem Height/Diameter: $R^2 > 0.98$.
  • Branch Traits: All extracted traits (angles, counts, diameters) showed an $R^2 > 0.8$ and a Mean Absolute Percentage Error (MAPE) of less than 10%.

Limitations and Future Work

  • The "Square" Problem: The model struggled to distinguish between mature cotton bolls and "squares" (small floral buds). Because they are geometrically similar, the current XYZ-only approach sometimes misclassifies them. The authors suggest that adding RGB data back into the mix might help solve this.
  • Dataset Scaling: While 30 plants provided a strong proof-of-concept, more genotypes and growth stages are needed to ensure the model generalizes across all cotton varieties.
  • Field Automation: The current workflow requires some manual denoising in software like CloudCompare. The next step is to fully automate the preprocessing to allow for "scan-to-trait" pipelines in the field.

Source Code: The authors have made their segmentation code available on GitHub at UGA-BSAIL/plant_3d_deep_learning.

Summary updated

Abstract

This plant part segmentation method based on 3D deep learning enables effective and efficient architectural trait measurement from point clouds, which could be useful to advance plant breeding programs and characterization of in-season developmental traits. The plant part segmentation code is available at https://github.com/UGA-BSAIL/plant_3d_deep_learning .
BibTeX
@inproceedings{saeed2023cotton,
  author = {Farah Saeed and Shangpeng Sun and Javier Rodriguez-Sanchez and John L. Snider and Tianming Liu and Changying Li},
  title = {{Cotton plant part 3D segmentation and architectural trait extraction using point voxel convolutional neural networks}},
  booktitle = {Plant},
  year = {2023},
  doi = {10.1186/s13007-023-00996-1},
}
Figure