Conifer Seedling Detection in UAV-Imagery with RGB-Depth Information

Jason Jooste, Michael Fromm, Matthias SchubertarXiv 2021 · arXiv

Introduction

Reforestation is a critical component of environmental restoration, particularly in the boreal forests of Alberta, Canada. In these regions, seismic exploration for oil and gas has created thousands of kilometers of narrow clearings that require monitoring to ensure successful regrowth. Traditionally, counting and measuring seedlings is a labor-intensive manual task. While Unmanned Aerial Vehicles (UAVs) and object detection models have streamlined this process, conifer seedlings remain difficult to detect due to their small size and tendency to blend into surrounding vegetation like moss or grass.

This paper demonstrates that integrating height (elevation) data with standard RGB drone imagery significantly improves detection accuracy. Crucially, the researchers found that the method of integration is vital: while adding depth as a fourth image channel (early fusion) is ineffective, injecting it after the feature-extraction backbone (mid-fusion) provides a substantial performance boost.

Methodology

The researchers employed a Faster R-CNN architecture with a ResNet50-FPN backbone. They investigated three primary strategies for fusing height data with visual features:

  1. Early Fusion ("First"): Height is added as a fourth channel (RGB-D) at the input layer.
  2. Late Fusion ("Final"): Height data is concatenated to features only at the final classification and bounding-box regression stage.
  3. Mid-Fusion ("Pre-RPN"): Height information is concatenated to the feature maps after the backbone but before the Region Proposal Network (RPN) and Region of Interest (RoI) heads.
Proposed fusion architectures within the Faster R-CNN framework.

The study utilized a custom dataset of 3,865 image tiles (256x256 pixels) captured at a 30m altitude, yielding a Ground Sampling Distance (GSD) of ~0.75cm. Height data was derived from photogrammetric Digital Surface Models (DSM) and, for specific areas, high-resolution LiDAR-based Canopy Height Models (CHM).

Key Results and Insights

The experimental results highlighted a significant "Backbone Bottleneck." The "First" model (early fusion) often performed worse than the baseline, likely because the ResNet backbone, pre-trained on 3-channel RGB data, could not effectively process the additional depth channel.

In contrast, the Pre-RPN model emerged as the superior architecture. By bypassing the backbone, the geometric height information remains distinct and provides immediate spatial context to the RPN, facilitating better candidate proposals.

Performance comparison (mAP) across different fusion strategies and experimental setups.

As shown in the results table, the Pre-RPN model achieved a Mean Average Precision (MAP) of 0.612 without pre-training, compared to 0.289 for the vanilla RGB model. With transfer learning (pre-training on COCO), the Pre-RPN model's MAP rose to 0.704. Further optimization of the feature map depth for height information (setting it to 32) resulted in a peak MAP of 0.721.

Interestingly, increasing the resolution of height data (comparing SD to HD LiDAR) provided only marginal gains. This suggests that even relatively coarse elevation data is sufficient to help the model distinguish 3D seedlings from the 2D forest floor.

Limitations and Future Work

Despite the improvements, several challenges remain. The current dataset is relatively small, and future work should explore extensive data augmentation to mitigate overfitting. Additionally, while 256x256 tiles were used, larger input sizes might improve Faster R-CNN performance at a higher computational cost.

The authors also suggest investigating "Super-resolution" or depth-estimation networks to predict height from RGB images when true elevation data is unavailable. Finally, addressing class imbalance—specifically the drop in performance when many "negative" (seedling-free) tiles are included—is essential for real-world deployment.

Summary updated

Abstract

Monitoring of reforestation is currently being considerably streamlined through the use of drones and image recognition algorithms, which have already proven to be effective on colour imagery. In addition to colour imagery, elevation data is often also available. The primary aim of this work was to improve the performance of the faster-RCNN object detection algorithm by integrating this height information, which showed itself to notably improve performance. Interestingly, the structure of the network played a key role, with direct addition of the height information as a fourth image channel showing no improvement, while integration after the backbone network and before the region proposal network led to marked improvements. This effect persisted with very long training regimes. Increasing the resolution of this height information also showed little effect.
BibTeX
@misc{jooste2021conifer,
  author = {Jason Jooste and Michael Fromm and Matthias Schubert},
  title = {{Conifer Seedling Detection in UAV-Imagery with RGB-Depth Information}},
  year = {2021},
  eprint = {2111.11388},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.11388},
}
Figure