Plant stem and leaf segmentation and phenotypic parameter extraction using neural radiance fields and lightweight point cloud segmentation networks

Guibin Qiao, Zhibin Zhang, Bin Niu, Sijia Han, En‐hui YangFrontiers in Plant Science 2025 · DOI

TL;DR

This paper introduces an end-to-end pipeline for automated plant phenotyping that combines Neural Radiance Fields (NeRF) for high-fidelity 3D reconstruction with a novel, lightweight segmentation network called PointSegNet. By extracting dense point clouds from NeRF models and applying edge-aware deep learning, the researchers achieved highly accurate measurements of maize traits—like leaf length and stem height—outperforming much heavier state-of-the-art models.

What Problem Does This Paper Address?

Accurately measuring a plant's physical traits (phenotyping) is essential for crop breeding and smart agriculture. Traditionally, this is done manually—a process that is slow, labor-intensive, and potentially damaging to the plants.

While computer vision has moved toward 2D image-based automation, plants are complex 3D structures. 2D images suffer from "occlusion" (leaves hiding stems or other leaves), leading to significant measurement errors. 3D sensors like LiDAR are precise but expensive and often struggle with the fine, overlapping geometries of foliage. The authors sought a way to use accessible hardware (like smartphones) to generate high-quality 3D data and a computationally efficient way to process that data into meaningful biological parameters.

Methods

The researchers developed a three-stage workflow:

  1. 3D Reconstruction via Nerfacto: Instead of traditional photogrammetry, they used Nerfacto (a variant of Neural Radiance Fields). NeRFs represent a scene as a continuous volumetric function. By taking a short video around a plant with a smartphone, the system learns an implicit 3D representation from which a dense, high-quality point cloud can be extracted.
  2. PointSegNet (The Core Innovation): The authors proposed a new, lightweight point cloud segmentation network. It features two key modules:
  3. Global-Local Set Abstraction (GLSA): This allows the network to understand both the fine details of a leaf's surface and the overall structure of the plant simultaneously.
  4. Edge-Aware Feature Propagation (EAFP): This module specifically targets the boundaries between organs (e.g., where a leaf meets a stem), which are notoriously difficult for standard networks to segment accurately.
  5. PCA-Based Trait Extraction: Once the point cloud is segmented into "leaf" and "stem," the system uses Principal Component Analysis (PCA) to determine the primary growth directions. This allows for the calculation of curved leaf lengths and widths, rather than just simple straight-line distances.

Datasets

The study utilized three primary datasets to train and validate the model:

  • Maize Dataset: A new dataset created by the authors consisting of 20 maize plants at various growth stages, expanded through data augmentation to 448 samples.
  • Pheno4D (Tomato): A public dataset used to test the model’s ability to handle the more complex, branching structures of tomato plants.
  • Soybean-MVS: A public dataset used to validate generalization across different species.

Key Insights and Ideas

  • Efficiency Matters: PointSegNet achieves superior accuracy with only 1.33 million parameters. Compared to other models like SPoTr (25.56M) or PointMLP (16.71M), it is significantly more "lightweight," making it a candidate for deployment on edge devices or mobile platforms in the field.
  • Implicit vs. Explicit: By using NeRFs (implicit) rather than just raw stereo matching (explicit), the system handles thin structures and overlapping leaves much better than traditional 3D reconstruction methods like Colmap or 3DF Zephyr.
  • The "Edge" Case: Most segmentation errors in plant phenotyping occur at the junctions of organs. The inclusion of the Edge-Aware (EAFP) module was the "aha moment" that allowed the model to maintain smooth, realistic boundaries even in dense foliage.

Key Results

The results demonstrate that PointSegNet is both faster and more accurate than current benchmarks:

  • Segmentation Accuracy: On maize, the model reached a 93.73% mIoU (mean Intersection over Union), surpassing PointNet++, PCT, and PointVector.
  • Phenotypic Precision: The extracted parameters showed a very high correlation with manual measurements:
  • Stem Height: $R^2 = 0.99$
    • Leaf Length: $R^2 = 0.94$
    • Stem Diameter: $R^2 = 0.84$
    • Leaf Width: $R^2 = 0.87$
  • Generalization: The model also achieved the highest metrics on the tomato and soybean datasets, proving it isn't just "overfit" to the geometry of maize.

Limitations and Future Work

While the pipeline is powerful, the authors noted a few hurdles:

  • Reconstruction Time: Training a NeRF model still takes 20–30 minutes per plant on a consumer GPU. While faster than some methods, it isn't yet "real-time."
  • Lighting Sensitivity: High-quality reconstruction still depends on consistent environmental lighting; shadows or glare during video capture can introduce noise into the point cloud.
  • Point Density: To keep the network lightweight, point clouds were downsampled to 2048 points. For extremely complex plants, this might result in the loss of very fine structural details.

The team plans to explore CUDA programming to further accelerate the reconstruction and investigate semi-supervised learning to reduce the need for manually labeled training data.

Summary updated

Abstract

High-quality 3D reconstruction and accurate 3D organ segmentation of plants are crucial prerequisites for automatically extracting phenotypic traits. In this study, we first extract a dense point cloud from implicit representations, which derives from reconstructing the maize plants in 3D by using the Nerfacto neural radiance field model. Second, we propose a lightweight point cloud segmentation network (PointSegNet) specifically for stem and leaf segmentation. This network includes a Global-Local Set Abstraction (GLSA) module to integrate local and global features and an Edge-Aware Feature Propagation (EAFP) module to enhance edge-awareness. Experimental results show that our PointSegNet achieves impressive performance compared to five other state-of-the-art deep learning networks, reaching 93.73%, 97.25%, 96.21%, and 96.73% in terms of mean Intersection over Union (mIoU), precision, recall, and F1-score, respectively. Even when dealing with tomato and soybean plants, with complex structures, our PointSegNet also achieves the best metrics. Meanwhile, based on the principal component analysis (PCA), we further optimize the method to obtain the parameters such as leaf length and leaf width by using PCA principal vectors. Finally, the maize stem thickness, stem height, leaf length, and leaf width obtained from our measurements are compared with the manual test results, yielding <i>R</i> <sup>2</sup> values of 0.99, 0.84, 0.94, and 0.87, respectively. These results indicate that our method has high accuracy and reliability for phenotypic parameter extraction. This study throughout the entire process from 3D reconstruction of maize plants to point cloud segmentation and phenotypic parameter extraction, provides a reliable and objective method for acquiring plant phenotypic parameters and will boost plant phenotypic development in smart agriculture.
BibTeX
@inproceedings{qiao2025plant,
  author = {Guibin Qiao and Zhibin Zhang and Bin Niu and Sijia Han and En‐hui Yang},
  title = {{Plant stem and leaf segmentation and phenotypic parameter extraction using neural radiance fields and lightweight point cloud segmentation networks}},
  booktitle = {Frontiers in Plant Science},
  year = {2025},
  doi = {10.3389/fpls.2025.1491170},
}
Figure