Abstract
In this study, we propose a high-throughput and low-cost automatic detection method based on deep learning to replace the inefficient manual counting of rapeseed siliques. First, a video is captured with a smartphone around the rapeseed plants in the silique stage. Feature point detection and matching based on SIFT operators are applied to the extracted video frames, and sparse point clouds are recovered using epipolar geometry and triangulation principles. The plant model of the whole rapeseed plant is reconstructed based on the structure-from-motion (SfM) algorithm. The downsampled 3D point cloud data is processed by a Dynamic Graph CNN (DGCNN) network to classify points into siliques and stems. These labels are then mapped back to the original dense point cloud using a sparse-dense mapping method to improve efficiency. Finally, Euclidean clustering and the RANSAC algorithm are used to segment and count individual siliques. The proposed method identified 1,457 siliques from 12 plants with a recognition accuracy greater than 97.80%.
The Challenge of Rapeseed Phenotyping
In rapeseed breeding and yield estimation, the number of siliques (seed pods) per plant is the most critical metric. Traditionally, this data is collected via manual counting—a process that is agonizingly slow, labor-intensive, and prone to human error. While 2D computer vision methods exist, they frequently struggle with "occlusion," where pods overlap and hide each other in a flat image.
3D reconstruction offers a solution by capturing the plant's full geometry, but it usually requires expensive LiDAR equipment. Even when using cheaper "Structure from Motion" (SfM) techniques to build 3D models from photos, the resulting "point clouds" contain millions of data points. Processing these dense clouds through deep learning models is often too computationally expensive for practical, large-scale use.
Methodology: The Sparse-Dense Mapping Pipeline
The researchers developed a multi-stage pipeline designed to turn a 30-second smartphone video into a precise count while maintaining computational efficiency.
1. 3D Reconstruction and Pruning
To avoid processing hundreds of redundant video frames, the system uses a Laplacian variance check to remove blurry frames and SURF-based feature matching to filter out nearly identical images. The remaining frames are processed using the SfM algorithm to reconstruct a dense 3D point cloud of the plant.
2. DGCNN Segmentation and Label Propagation
The core innovation lies in the "Sparse-Dense Mapping" trick. Instead of feeding a massive dense cloud into a deep learning model, the cloud is downsampled to a sparse version (roughly 8,192 points). A Dynamic Graph CNN (DGCNN) classifies these sparse points into "silique" or "stem." Unlike standard CNNs that look at fixed grids, DGCNNs analyze the relationships (edges) between points, which is ideal for capturing the long, slender geometry of a rapeseed pod.
Using a Kd-tree radius search, the system then maps the labels from the sparse points back onto the original dense cloud. This provides the high resolution needed for accurate counting without the high cost of segmenting every single point in the dense cloud.
3. Individual Counting
Finally, the system uses Euclidean clustering to group points into individual pods. To handle pods that are touching or overlapping, the RANSAC algorithm fits straight lines to the clusters, ensuring that even adhered siliques are counted as separate entities.
Key Results and Performance
The study evaluated the system using three rapeseed varieties (Zhongshuang 6, Dadi 55, and Huayouza 62). The experimental results demonstrated the high precision of the approach:
- High Accuracy: The system identified 1,425 out of 1,457 siliques, achieving a recognition accuracy of 97.80%.
- Statistical Reliability: The mean absolute percentage error was only 1.96%, with a coefficient of determination ($R^2$) of 0.96, indicating a very high correlation between automated counts and ground truth.
- Architectural Superiority: DGCNN outperformed other popular 3D architectures like PointNet and PointNet++ across all metrics, including Precision, Recall, and Intersection over Union (IoU).
Conclusion and Future Directions
By proving that a standard smartphone can produce research-grade phenotyping data, this method lowers the barrier to entry for digital agriculture. While the segmentation is fast, the initial SfM reconstruction remains a bottleneck, often taking over 30 minutes per plant. Future work will focus on GPU acceleration for the SfM stage and adapting the system for "wild" field environments where wind and complex backgrounds present additional challenges.
Summary updated