Panicle-3D: Efficient Phenotyping Tool for Precise Semantic Segmentation of Rice Panicle Point Cloud

Liang Gong, Xiaofeng Du, Kai Zhu, Ke Lin, Qiaojun Lou, Zheng Yuan, Guoqiang Huang, Chengliang LiuPlant 2021 · DOI

TL;DR

This paper introduces Panicle-3D, a novel deep learning model for precise semantic segmentation of rice panicle point clouds. By developing a custom 3D scanning platform and a U-Net-like architecture incorporating PointConv and skip connections, they achieve state-of-the-art segmentation accuracy, enabling automated phenotyping for crop research.

What Problem Does This Paper Address?

Accurately measuring crop phenotypic parameters is crucial for understanding plant growth, genetics, and yield. Traditional methods often rely on manual measurements, which are time-consuming and labor-intensive. While 3D point cloud segmentation offers a path to automation, segmenting complex plant structures like rice panicles presents significant challenges. These include a scarcity of labeled data, uneven point cloud distribution, occlusion from leaves and stems, and the inherent disorder of point cloud data, which traditional algorithms struggle to handle effectively. Existing deep learning approaches for point clouds, like PointNet, have limitations in capturing local relationships and are not specifically tailored for the multiscale characteristics of plant organs.

Methods

The authors developed a desktop-level, structured-light 3D scanning platform to acquire high-quality point cloud data of rice panicles. This platform utilizes a Digital Light Processing (DLP) projector and a rotating platform for comprehensive 3D reconstruction.

For data preprocessing, they employed manual outlier removal, Gaussian filtering for noise reduction, and farthest point sampling to downsample the point clouds to a uniform size suitable for neural network input. Crucially, they created a new dataset of labeled rice panicle point clouds, distinguishing between the rice stalk and the rice ear. Data augmentation techniques were applied to increase sample diversity and improve model robustness.

The core of their contribution is the Panicle-3D neural network. This model is a U-Net-like Fully Convolutional Network (FCN) designed for semantic segmentation. Key architectural features include:

  • PointConv: A convolution operation specifically designed for irregular point clouds, addressing the computational intensity of voxel-based or image-based 3D convolutions.
  • SE-Inception-PointConv Module: This module integrates the Inception structure for capturing features at multiple scales and the Squeeze-and-Excitation (SE) network for channel-wise feature recalibration, enhancing the model's ability to adapt to varying plant organ characteristics.
  • Long and Short Skip Connections: Inspired by U-Net and ResNet, these connections allow the model to fuse low-level and high-level semantic information, improving the segmentation of fine details and edges.

Datasets

The paper introduces a newly created dataset of rice panicle point clouds. This dataset was generated using their custom 3D scanning platform and manually annotated, distinguishing between rice stalks and rice ears. The dataset contains 200 rice point clouds. Data augmentation was applied to this dataset to improve training. The model was trained and evaluated on this dataset.

Key Insights and Ideas

  • The need for specialized hardware: Acquiring high-quality, dense 3D point cloud data of plant structures is a prerequisite for effective deep learning. The authors' custom scanner demonstrates a practical solution.
  • Data augmentation is critical for plant point clouds: Given the inherent complexity and difficulty in collecting large labeled datasets for plant structures, data augmentation is essential to prevent overfitting and improve model generalization.
  • Multiscale feature extraction is key for plant organ segmentation: Plant organs exhibit significant variations in scale and structure. The SE-Inception-PointConv module effectively addresses this by combining multi-scale receptive fields (Inception) with channel attention (SE-Net).
  • Skip connections are vital for detail preservation: The combination of long (U-Net-like) and short (ResNet-like) skip connections significantly improves the model's ability to capture fine details and accurately delineate the boundaries of plant organs.

Key Results

The Panicle-3D model achieved a segmentation accuracy of 93.4% and an Intersection over Union (IoU) of 86.1% on the rice panicle point cloud dataset. This performance was superior to the classic PointNet model, which achieved an accuracy of 90% and an IoU of 84%. The ablation study demonstrated that using both long and short skip connections yielded the lowest training and validation loss, highlighting the importance of this architectural choice.

Limitations and Future Work

The authors acknowledge several limitations:

  • Data scarcity and labor-intensiveness: Collecting and labeling 3D point cloud data for plants is a time-consuming and labor-intensive process.
  • Occlusion by leaves: When plant organs are heavily occluded by leaves, accurate labeling of the ground truth becomes challenging, potentially impacting model performance.
  • Destructive annotation: In some cases, to accurately label hidden parts of the panicle, a destructive approach (detaching grains) might be necessary for ground truth generation.

Future work could focus on developing more automated annotation tools and exploring methods to handle severe occlusion more effectively. The authors suggest that Panicle-3D is suitable for segmenting other similar crop point cloud tasks, indicating potential for broader application in plant phenotyping.

Summary updated

Abstract

The automated measurement of crop phenotypic parameters is of great significance to the quantitative study of crop growth. The segmentation and classification of crop point cloud help to realize the automation of crop phenotypic parameter measurement. At present, crop spike-shaped point cloud segmentation has problems such as fewer samples, uneven distribution of point clouds, occlusion of stem and spike, disorderly arrangement of point clouds, and lack of targeted network models. The traditional clustering method can realize the segmentation of the plant organ point cloud with relatively independent spatial location, but the accuracy is not acceptable. This paper first builds a desktop-level point cloud scanning apparatus based on a structured-light projection module to facilitate the point cloud acquisition process. Then, the rice ear point cloud was collected, and the rice ear point cloud data set was made. In addition, data argumentation is used to improve sample utilization efficiency and training accuracy. Finally, a 3D point cloud convolutional neural network model called Panicle-3D was designed to achieve better segmentation accuracy. Specifically, the design of Panicle-3D is aimed at the multiscale characteristics of plant organs, combined with the structure of PointConv and long and short jumps, which accelerates the convergence speed of the network and reduces the loss of features in the process of point cloud downsampling. After comparison experiments, the segmentation accuracy of Panicle-3D reaches 93.4%, which is higher than PointNet. Panicle-3D is suitable for other similar crop point cloud segmentation tasks.
BibTeX
@inproceedings{gong2021panicle,
  author = {Liang Gong and Xiaofeng Du and Kai Zhu and Ke Lin and Qiaojun Lou and Zheng Yuan and Guoqiang Huang and Chengliang Liu},
  title = {{Panicle-3D: Efficient Phenotyping Tool for Precise Semantic Segmentation of Rice Panicle Point Cloud}},
  booktitle = {Plant},
  year = {2021},
  doi = {10.34133/2021/9838929},
}
Figure