A remote sensing derived data set of 100 million individual tree crowns for the National Ecological Observatory Network

Ben Weinstein, Sergio Marconi, Stephanie Bohlman, Alina Zare, Aditya Singh, Sarah Graves, Ethan P. White

TL;DR

This paper introduces a massive, open-source dataset of over 100 million individual tree crowns across 37 sites in the US, derived from remote sensing data using deep learning. This dataset addresses the long-standing challenge of obtaining individual tree information at broad scales, enabling new avenues for ecological research.

What Problem Does This Paper Address?

Understanding forest ecosystems at the individual tree level is crucial for assessing biodiversity, ecosystem services, and forestry applications. However, collecting this data manually across vast areas is prohibitively expensive and logistically challenging. While remote sensing technologies have advanced, translating raw sensor data into actionable information about individual trees has remained a significant technical hurdle. This gap limits our ability to conduct large-scale ecological analyses and compare forest dynamics across different regions.

Methods

The core of this work is a semi-supervised deep learning approach, building upon the "DeepForest" Python package. This workflow utilizes a one-shot object detector with a convolutional neural network backbone to identify tree crowns in RGB imagery. The model was pre-trained on ImageNet and then fine-tuned using a combination of weak labels derived from existing LiDAR tree detection algorithms and hand-annotated crowns from NEON sites. A key refinement in this study is the incorporation of a 3-meter height filter using LiDAR-derived canopy height models to improve accuracy and distinguish trees from smaller vegetation. The output for each detected crown includes its spatial location (bounding box), estimated crown area, an estimate of tree height, and a confidence score.

Datasets

  • Training Data:
    • ImageNet (for initial pre-training)
    • Weak labels generated from a previous LiDAR tree detection algorithm on 30 NEON sites.
  • 10,000 hand-annotated tree crowns from seven NEON sites.
  • 207 image-annotated bounding boxes (from 21 NEON sites) used for validation.
  • Evaluation Data:
    • NEON Airborne Observation Platform (AOP) data (RGB and LiDAR) from 37 NEON sites (2018 or 2019).
  • NEON Woody Vegetation Structure data (field-collected stem data).

The paper introduces a novel, large-scale dataset of individual tree crowns, which serves as a benchmark for future research in this area.

Key Insights and Ideas

  • Bridging Scales: The most significant insight is the successful creation of a dataset that bridges the gap between local, field-based measurements and broad-scale remote sensing, enabling individual-level analyses across continental extents.
  • Deep Learning's Power: The study demonstrates the efficacy of deep learning, specifically semi-supervised approaches, in tackling complex remote sensing tasks like individual tree crown detection, even with variations in data quality and forest types.
  • LiDAR as a Refinement: Incorporating LiDAR-derived height information proved crucial for improving the accuracy of tree detection, particularly in distinguishing trees from understory vegetation.
  • Generalizability: The model shows good generalization across diverse geographic sites and forest conditions, a critical factor for a continental-scale dataset.

Key Results

  • The dataset contains 104,675,304 individual tree crowns across 37 NEON sites.
  • The workflow achieved a bounding box recall of 79.1% and a precision of 72.6% when compared to image-annotated crowns.
  • Adding the LiDAR 3m height filter resulted in a 7.0% increase in precision with a slight decrease in recall (0.2%).
  • A linear-mixed model comparing predicted tree heights to field measurements yielded a root mean squared error (RMSE) of 1.73 m.
  • Overstory stem recall rate was 69.4% when compared to field-collected stem data.

Limitations and Future Work

  • Sun-Exposed Crowns Only: The dataset primarily captures sun-exposed canopy trees, excluding subcanopy trees. This limits its direct applicability to analyses heavily reliant on understory structure.
  • Undersegmentation Bias: The model tends to undersegment trees, grouping multiple individuals into a single crown, and may underestimate crown extent. This can impact studies of tree allometry and biomass.
  • Image-Based Annotations: The reliance on image-based annotations means that what appears as a single crown in imagery might be multiple trees, or vice-versa, introducing uncertainty.
  • Novel Object Detection: The model struggles with objects not present in the training data (e.g., buildings, weather stations), leading to potential false positives.
  • Data Fusion Challenges: Combining RGB and LiDAR data directly into the CNN did not yield improvements, possibly due to geographic variation, sparse LiDAR coverage, and lack of joint pre-training.

Future work could involve:

  • Developing methods to incorporate subcanopy tree information.
  • Improving the model's ability to handle undersegmentation and estimate crown extent more accurately.
  • Exploring techniques for detecting and labeling "unknown" objects.
  • Investigating advanced data fusion methods and leveraging hyperspectral data for improved species differentiation.
  • Iteratively improving the dataset with ongoing assessment and community annotation efforts.

Summary updated

Abstract

Forests provide biodiversity, ecosystem, and economic services. Information on individual trees is important for understanding forest ecosystems but obtaining individual-level data at broad scales is challenging due to the costs and logistics of data collection. While advances in remote sensing techniques allow surveys of individual trees at unprecedented extents, there remain technical challenges in turning sensor data into tangible information. Using deep learning methods, we produced an open-source data set of individual-level crown estimates for 100 million trees at 37 sites across the United States surveyed by the National Ecological Observatory Network’s Airborne Observation Platform. Each canopy tree crown is represented by a rectangular bounding box and includes information on the height, crown area, and spatial location of the tree. These data have the potential to drive significant expansion of individual-level research on trees by facilitating both regional analyses and cross-region comparisons encompassing forest types from most of the United States.
BibTeX
@inproceedings{weinstein2021remote,
  author = {Ben Weinstein and Sergio Marconi and Stephanie Bohlman and Alina Zare and Aditya Singh and Sarah Graves and Ethan P. White},
  title = {{A remote sensing derived data set of 100 million individual tree crowns for the National Ecological Observatory Network}},
  booktitle = {eLife},
  year = {2021},
  doi = {10.7554/elife.62922},
}
Figure