Improving High-Throughput Phenotyping Using Fusion of Close-Range Hyperspectral Camera and Low-Cost Depth Sensor

Peikui Huang, Xiwen Luo, Jian Xun Jin, Liangju Wang, Libo Zhang, Jie Liu, Zhigang ZhangSensors 2018 · DOI

TL;DR

This paper presents a novel method for fusing data from a close-range hyperspectral camera and a low-cost depth sensor to create detailed 3D plant models. This fusion overcomes limitations of hyperspectral imaging alone, enabling more accurate plant phenotyping by accounting for complex plant geometry and illumination effects.

What Problem Does This Paper Address?

High-throughput phenotyping (HTP) is crucial for accelerating crop breeding and addressing global food security. Hyperspectral imaging is a powerful HTP tool for detecting plant stress and biochemical properties. However, the complex 3D structure of plants (leaf angles, self-shading) and varying illumination conditions significantly distort spectral information, limiting its accuracy. Furthermore, critical plant traits like leaf area and inclination distribution are not directly captured by traditional hyperspectral sensors. Existing methods often average plant data, losing valuable spatial detail. This paper tackles the challenge of integrating spatial 3D information with spectral data to create more robust and informative plant models for HTP.

Methods

The core of this work is a novel fusion method that combines data from a close-range hyperspectral pushbroom camera and a low-cost Kinect V2 depth sensor. The approach involves:

  • Geometric Calibration: A key innovation is a simple and efficient geometric calibration method for the hyperspectral camera using a designed reference gauge with a checkerboard pattern. This method establishes the relationship between 3D world coordinates and 2D image coordinates.
  • Sensor Fusion Model: A fusion model is developed to align the hyperspectral image coordinates with the 3D point clouds from the depth sensor. This model accounts for the linear camera model of the pushbroom sensor and the movement of the imaging platform.
  • Semi-Automatic Coordinate Measurement: The algorithm uses a semi-automatic approach to detect reference points in the hyperspectral images, making it more flexible for partial or rotated views of the reference gauge.
  • 3D Point Cloud Processing: The Kinect V2 provides depth information, which is processed to generate 3D point clouds of the plant.
  • Data Alignment: The paper details the transformation matrices and equations used to align the coordinate systems of the hyperspectral camera, the depth sensor, and the base platform.

Datasets

The paper uses two main datasets for its experiments:

  • Reference Gauge Data: A custom-designed reference gauge with a checkerboard pattern was used for calibrating the hyperspectral camera and establishing the fusion model. This gauge provided a known set of 3D points and their corresponding 2D image projections.
  • Soybean Leaf Data: Soybean leaves (Asgrow 30X6 variety) were used for validation experiments. This involved capturing hyperspectral images and 3D point clouds of individual leaves under controlled conditions.

Key Insights and Ideas

  • Bridging the Spatial-Spectral Gap: The fusion of hyperspectral and depth data is essential for overcoming the limitations of spectral information alone, especially in complex plant structures.
  • Low-Cost Sensor Integration: Utilizing a low-cost depth sensor like Kinect V2 makes advanced 3D phenotyping more accessible and cost-effective.
  • Geometric Calibration is Key: A robust and simple geometric calibration method for the hyperspectral camera is crucial for accurate fusion.
  • 3D Reflectance Models: The generated 3D hyperspectral models open possibilities for creating 3D reflectance models, which can help minimize geometry-related artifacts in hyperspectral images.
  • Enhanced Phenotyping Precision: By incorporating 3D information, the method significantly improves the precision of biological process analysis in high-throughput phenotyping.

Key Results

  • High Accuracy: The proposed method achieved a 0.99 pixel resolution for the hyperspectral camera and a 3.3 millimeter accuracy for the depth sensor in a controlled environment.
  • Effective Fusion: Validation experiments with soybean leaves demonstrated the effectiveness of the fusion model, with a mean error of 2.2 mm and a maximum error of 3.3 mm in depth information derivation.
  • Quantitative Plant Parameters: The 3D hyperspectral models enabled the calculation of plant geometry parameters, such as leaf area (32.2 cm²) and inclination (63.1°) for a soybean leaf.

Limitations and Future Work

  • Controlled Environment: The experiments were conducted in a dark room with controlled illumination. Future work could explore the method's performance under variable natural lighting conditions.
  • Single Sensor Fusion: The current work focuses on fusing one hyperspectral camera and one depth sensor. The authors suggest extending this to fuse data from multiple hyperspectral cameras (e.g., top and side views) and other sensors (e.g., thermal imaging) to address overlapping issues and create more comprehensive 3D plant models.
  • Automation: While the calibration is semi-automatic, further automation of the entire process would be beneficial for large-scale HTP.
  • Illumination Modeling: The paper mentions the potential for 3D reflectance models to minimize illumination effects. Further research in this area could enhance robustness.
  • Plant Model Library: The creation of a large database of 3D plant models is proposed as a future direction, which could significantly improve simulation of illumination and analysis of phenotypic traits.

Summary updated

Abstract

Hyperspectral sensors, especially the close-range hyperspectral camera, have been widely introduced to detect biological processes of plants in the high-throughput phenotyping platform, to support the identification of biotic and abiotic stress reactions at an early stage. However, the complex geometry of plants and their interaction with the illumination, severely affects the spectral information obtained. Furthermore, plant structure, leaf area, and leaf inclination distribution are critical indexes which have been widely used in multiple plant models. Therefore, the process of combination between hyperspectral images and 3D point clouds is a promising approach to solve these problems and improve the high-throughput phenotyping technique. We proposed a novel approach fusing a low-cost depth sensor and a close-range hyperspectral camera, which extended hyperspectral camera ability with 3D information as a potential tool for high-throughput phenotyping. An exemplary new calibration and analysis method was shown in soybean leaf experiments. The results showed that a 0.99 pixel resolution for the hyperspectral camera and a 3.3 millimeter accuracy for the depth sensor, could be achieved in a controlled environment using the method proposed in this paper. We also discussed the new capabilities gained using this new method, to quantify and model the effects of plant geometry and sensor configuration. The possibility of 3D reflectance models can be used to minimize the geometry-related effects in hyperspectral images, and to significantly improve high-throughput phenotyping. Overall results of this research, indicated that the proposed method provided more accurate spatial and spectral plant information, which helped to enhance the precision of biological processes in high-throughput phenotyping.
BibTeX
@inproceedings{huang2018improving,
  author = {Peikui Huang and Xiwen Luo and Jian Xun Jin and Liangju Wang and Libo Zhang and Jie Liu and Zhigang Zhang},
  title = {{Improving High-Throughput Phenotyping Using Fusion of Close-Range Hyperspectral Camera and Low-Cost Depth Sensor}},
  booktitle = {Sensors},
  year = {2018},
  doi = {10.3390/s18082711},
}
Figure