TL;DR
This paper introduces a novel vision-based system for automated plant phenotyping in vertical farms. By combining 3D modeling and deep learning on RGBD images, the system accurately predicts plant height, leaf area, and leaf weight, offering a non-destructive and scalable solution for optimizing crop growth in controlled environments.
What Problem Does This Paper Address?
The growing demand for fresh, locally sourced food, coupled with environmental concerns, is driving the expansion of vertical farming. However, optimizing crop yield and quality in these controlled environments requires precise monitoring of plant growth. Traditional phenotyping methods are labor-intensive and time-consuming, hindering scalability. This paper addresses the need for an automated, accurate, and non-destructive method to track plant development in vertical farms, specifically under artificial lighting (VFAL), where environmental conditions are stable but plant monitoring can be challenging due to stacked layers and limited accessibility.
Methods
The core of this work lies in a vision-based pipeline that leverages RGBD cameras to capture detailed 3D information of plants. The process involves:
- RGBD Image Acquisition: Using commercial RGBD sensors to capture both color and depth information of plants at various growth stages (DAS).
- 3D Reconstruction and Noise Removal: Converting depth maps into point clouds and applying noise reduction techniques, including plane fitting and Iterative Closest Point (ICP) algorithm, to create a clean 3D representation.
- Plant Height Prediction: Developing a method to accurately measure plant height by fitting lines to tray cells in early-stage images and using the depth map to derive height surfaces. This approach accounts for plant density and aims to measure the highest plant within each plug.
- Leaf Segmentation and Area/Weight Estimation: Employing Mask R-CNN for instance segmentation of visible leaves. The segmented masks are projected onto the depth map to reconstruct leaf geometry. Leaf area is then calculated from these reconstructions. Leaf weight is derived using a pre-established density relationship (PBL) or through a nonlinear regression model trained on a small dataset of leaves with known area and weight.
- Machine Learning for Prediction: While the paper focuses on vision-based measurements, it also mentions using a fully connected neural network to predict leaf area from leaf weight, demonstrating the potential for integrating different data sources.
Datasets
- Newly Introduced Benchmark: The paper introduces a novel dataset collected from three experimental cycles of basil growth in a vertical farm. This dataset comprises:
- 2592 manually collected data points for plant phenotype (height, leaf weight, number of leaves, plug weight).
- 1728 images (RGBD) of the plants captured daily.
- Existing Datasets (for training/benchmarking):
- COCO dataset (mentioned as not directly usable for training due to domain differences).
- Publicly available Mask R-CNN implementation in Tensorflow.
- LabelMe for manual annotation of leaf samples.
Key Insights and Ideas
- VFAL as an Ideal Environment for Vision-Based Phenotyping: The stable and controlled environment of VFALs, free from external variations like solar intensity, makes them particularly well-suited for precise vision-based monitoring.
- 3D Reconstruction for Accurate Measurement: The use of RGBD cameras and subsequent 3D reconstruction is crucial for obtaining accurate measurements of plant height and leaf area, overcoming the limitations of 2D imaging.
- Plant Density Impacts Growth Patterns: The study highlights how plant density influences height and its standard deviation, with denser plants exhibiting more stabilized growth patterns due to competition.
- Non-Destructive Leaf Area Estimation: The ability to estimate leaf area from visible leaves, even with occlusions, is a significant step towards non-destructive yield prediction.
- Vision-Based Methods Complement Manual Measurements: The vision-based approach provides a high-resolution, continuous stream of data that complements and, in some aspects, surpasses the accuracy and detail of manual measurements.
Key Results
- Accurate Height Prediction: The vision-based method for plant height prediction demonstrated high accuracy, closely matching manual measurements. The Bhattacharyya distance was used to quantify the error between predicted and measured height distributions.
- Leaf Area and Weight Prediction: The system successfully predicted leaf area and, subsequently, leaf weight. The accuracy of leaf area prediction was validated by inverting the network to recompute weights, showing limited error.
- Data Collection: The paper successfully collected a substantial dataset (2592 manual measurements, 1728 images) for training and validation.
- Feature Discrimination: Plant height, leaf area, and leaf weight were shown to be effective features for discriminating between high and low plant densities.
Limitations and Future Work
- Leaf Area Occlusion: The current leaf segmentation method can only measure leaf area for surfaces not occluded by more than 60% by other leaves.
- Small Training Set for Leaf Segmentation: The limited number of labeled leaf samples (270 large, 180 small) for training the Mask R-CNN model is a constraint.
- Future Work: The authors suggest extending this work to obtain quantitative information on plant phenotype, nutrition values, and even taste using vision-based techniques. They also aim to understand the effect of various LED light parameters, environmental conditions, and grow conditions on crop growth behavior. The ultimate goal is to maximize growth and quality while minimizing resource use and to develop non-destructive methods for assessing nutritional values and crop quality.
Summary updated