Point clouds segmentation of rapeseed siliques based on sparse-dense point clouds mapping

Yuhui Qiao, Qingxi Liao, Moran Zhang, Binbin Han, Chengli Peng, Zhenhao Huang, Shaodong Wang, Guangsheng Zhou, Shengyong XuFrontiers in Plant Science 2023 · DOI

Abstract

In this study, we propose a high-throughput and low-cost automatic detection method based on deep learning to replace the inefficient manual counting of rapeseed siliques. First, a video is captured with a smartphone around the rapeseed plants in the silique stage. Feature point detection and matching based on SIFT operators are applied to the extracted video frames, and sparse point clouds are recovered using epipolar geometry and triangulation principles. The plant model of the whole rapeseed plant is reconstructed based on the structure-from-motion (SfM) algorithm. The downsampled 3D point cloud data is processed by a Dynamic Graph CNN (DGCNN) network to classify points into siliques and stems. These labels are then mapped back to the original dense point cloud using a sparse-dense mapping method to improve efficiency. Finally, Euclidean clustering and the RANSAC algorithm are used to segment and count individual siliques. The proposed method identified 1,457 siliques from 12 plants with a recognition accuracy greater than 97.80%.

The Challenge of Rapeseed Phenotyping

In rapeseed breeding and yield estimation, the number of siliques (seed pods) per plant is the most critical metric. Traditionally, this data is collected via manual counting—a process that is agonizingly slow, labor-intensive, and prone to human error. While 2D computer vision methods exist, they frequently struggle with "occlusion," where pods overlap and hide each other in a flat image.

3D reconstruction offers a solution by capturing the plant's full geometry, but it usually requires expensive LiDAR equipment. Even when using cheaper "Structure from Motion" (SfM) techniques to build 3D models from photos, the resulting "point clouds" contain millions of data points. Processing these dense clouds through deep learning models is often too computationally expensive for practical, large-scale use.

Methodology: The Sparse-Dense Mapping Pipeline

The researchers developed a multi-stage pipeline designed to turn a 30-second smartphone video into a precise count while maintaining computational efficiency.

Overview of the proposed rapeseed silique detection and counting pipeline, including 3D reconstruction, DGCNN segmentation, sparse-dense mapping, and individual silique identification.

1. 3D Reconstruction and Pruning

To avoid processing hundreds of redundant video frames, the system uses a Laplacian variance check to remove blurry frames and SURF-based feature matching to filter out nearly identical images. The remaining frames are processed using the SfM algorithm to reconstruct a dense 3D point cloud of the plant.

2. DGCNN Segmentation and Label Propagation

The core innovation lies in the "Sparse-Dense Mapping" trick. Instead of feeding a massive dense cloud into a deep learning model, the cloud is downsampled to a sparse version (roughly 8,192 points). A Dynamic Graph CNN (DGCNN) classifies these sparse points into "silique" or "stem." Unlike standard CNNs that look at fixed grids, DGCNNs analyze the relationships (edges) between points, which is ideal for capturing the long, slender geometry of a rapeseed pod.

Using a Kd-tree radius search, the system then maps the labels from the sparse points back onto the original dense cloud. This provides the high resolution needed for accurate counting without the high cost of segmenting every single point in the dense cloud.

3. Individual Counting

Finally, the system uses Euclidean clustering to group points into individual pods. To handle pods that are touching or overlapping, the RANSAC algorithm fits straight lines to the clusters, ensuring that even adhered siliques are counted as separate entities.

Key Results and Performance

The study evaluated the system using three rapeseed varieties (Zhongshuang 6, Dadi 55, and Huayouza 62). The experimental results demonstrated the high precision of the approach:

  • High Accuracy: The system identified 1,425 out of 1,457 siliques, achieving a recognition accuracy of 97.80%.
  • Statistical Reliability: The mean absolute percentage error was only 1.96%, with a coefficient of determination ($R^2$) of 0.96, indicating a very high correlation between automated counts and ground truth.
  • Architectural Superiority: DGCNN outperformed other popular 3D architectures like PointNet and PointNet++ across all metrics, including Precision, Recall, and Intersection over Union (IoU).

Conclusion and Future Directions

By proving that a standard smartphone can produce research-grade phenotyping data, this method lowers the barrier to entry for digital agriculture. While the segmentation is fast, the initial SfM reconstruction remains a bottleneck, often taking over 30 minutes per plant. Future work will focus on GPU acceleration for the SfM stage and adapting the system for "wild" field environments where wind and complex backgrounds present additional challenges.

Summary updated

Abstract

In this study, we propose a high-throughput and low-cost automatic detection method based on deep learning to replace the inefficient manual counting of rapeseed siliques. First, a video is captured with a smartphone around the rapeseed plants in the silique stage. Feature point detection and matching based on SIFT operators are applied to the extracted video frames, and sparse point clouds are recovered using epipolar geometry and triangulation principles. The depth map is obtained by calculating the disparity of the matched images, and the dense point cloud is fused. The plant model of the whole rapeseed plant in the silique stage is reconstructed based on the structure-from-motion (SfM) algorithm, and the background is removed by using the passthrough filter. The downsampled 3D point cloud data is processed by the DGCNN network, and the point cloud is divided into two categories: sparse rapeseed canopy siliques and rapeseed stems. The sparse canopy siliques are then segmented from the original whole rapeseed siliques point cloud using the sparse-dense point cloud mapping method, which can effectively save running time and improve efficiency. Finally, Euclidean clustering segmentation is performed on the rapeseed canopy siliques, and the RANSAC algorithm is used to perform line segmentation on the connected siliques after clustering, obtaining the three-dimensional spatial position of each silique and counting the number of siliques. The proposed method was applied to identify 1457 siliques from 12 rapeseed plants, and the experimental results showed a recognition accuracy greater than 97.80%. The proposed method achieved good results in rapeseed silique recognition and provided a useful example for the application of deep learning networks in dense 3D point cloud segmentation.
BibTeX
@inproceedings{qiao2023point,
  author = {Yuhui Qiao and Qingxi Liao and Moran Zhang and Binbin Han and Chengli Peng and Zhenhao Huang and Shaodong Wang and Guangsheng Zhou and Shengyong Xu},
  title = {{Point clouds segmentation of rapeseed siliques based on sparse-dense point clouds mapping}},
  booktitle = {Frontiers in Plant Science},
  year = {2023},
  doi = {10.3389/fpls.2023.1188286},
}
Figure