The Auto Arborist Dataset: A Large-Scale Benchmark for Multiview Urban Forest Monitoring Under Domain Shift

Sara Beery, Guanhang Wu, Trevor Edwards, Filip Pavetic, Bo Majewski, Shreyasee Mukherjee, Stanley Chan, John Morgan, Vivek Rathod, Jonathan Huang

Introduction: The Critical Need for Urban Forest Monitoring

Urban forests are essential infrastructure, providing cooling, carbon sequestration, and significant public health benefits. However, maintaining an accurate tree census is a massive logistical and financial challenge. For instance, a single census in Los Angeles can cost upwards of $2 million and take years to complete. Consequently, many urban forest records are outdated or non-existent, particularly in under-resourced areas.

Computer vision (CV) offers a potential solution for automated monitoring, but existing datasets have historically been limited in scale or geographic scope. The primary technical hurdles include geographic domain shift—where models fail to generalize across different cities due to variations in architecture, lighting, and local flora—and the long-tailed distribution of tree species, where a few common genera dominate the data while hundreds of others are rarely seen.

The Auto Arborist Dataset: Scale and Diversity

To address these challenges, the authors introduce the Auto Arborist Dataset, a benchmark that is over two orders of magnitude larger than previous urban forest datasets. It consists of 2.6 million tree records across 23 North American cities, spanning 344 unique genera.

Geographic coverage of the Auto Arborist dataset across North America, featuring paired aerial and multiview street-level imagery.

The dataset provides a multimodal perspective for each tree, merging public census data with approximately 9.1 million images. This includes 512x512 aerial crops and up to three 768x1152 street-level panoramas per tree. This multiview approach allows models to capture both the canopy structure from above and the trunk/leaf details from the side.

Methodology: Multimodal Fusion and Regional Evaluation

The researchers developed a pipeline using ResNet-101 backbones to process both aerial and street-level imagery. To effectively combine these modalities, they employed a Mixture of Experts (MoE) approach, enabling the model to dynamically weigh the importance of aerial versus street-level data based on the specific genus or geographic region.

Evaluation was conducted hierarchically to test generalization:

  1. Per-city: Learning within a single urban environment.
  2. Regional: Generalizing across broad geographic areas (West, Central, East).
  3. Full Dataset: Testing a global model on entirely unseen holdout cities.

Key Insights: Domain Shift and the Long Tail

The study revealed several critical findings regarding urban tree classification:

  • Street-level Dominance: Street-level imagery is the most informative modality, providing a >20% boost in Class-Averaged Recall (AR) compared to aerial data alone.
  • Regional Generalization: Models generalize significantly better to new cities within the same geographic region. This suggests that local "terroir"—including climate and urban planning styles—creates a distinct visual signature that models can learn.
  • The Challenge of Rarity: Despite the massive scale of the dataset, rare genera (those with fewer than 100 examples) remain difficult to classify accurately, highlighting the need for improved few-shot learning.
Distribution of tree genera in the training and test sets, highlighting the extreme long-tail nature of urban forest data.

Results and Future Directions

The best-performing model, utilizing the full dataset and a regional MoE, achieved a 49.96% Class-Averaged Recall (AR). While top-1 accuracy reached as high as 93.28% in some cities (e.g., Vancouver), the variance across cities highlights the difficulty of handling diverse local distributions. Moving from a single street-level view to a multiview + aerial approach provided a consistent performance boost of 3-5%.

Future work will focus on addressing label noise from public censuses, moving from genus-level to species-level identification, and improving privacy-preserving techniques for large-scale street-level imagery. The ultimate goal is a "virtual census" capability that can monitor urban forests globally using only available remote sensing data.

Summary updated

Abstract

Generalization to novel domains is a fundamental chal-lenge for computer vision. Near-perfect accuracy on bench-marks is common, but these models do not work as expected when deployed outside of the training distribution. To build computer vision systems that truly solve real-world prob-lems at global scale, we need benchmarks that fully capture real-world complexity, including geographic domain shift, long-tailed distributions, and data noise. We propose urban forest monitoring as an ideal testbed for studying and improving upon these computer vision challenges, while working towards filling a crucial environ-mental and societal need. Urban forests provide significant benefits to urban societies. However, planning and main-taining these forests is expensive. One particularly costly aspect of urban forest management is monitoring the ex-isting trees in a city: e.g., tracking tree locations, species, and health. Monitoring efforts are currently based on tree censuses built by human experts, costing cities millions of dollars per census and thus collected infrequently. Previous investigations into automating urban forest monitoring focused on small datasets from single cities, covering only common categories. To address these short-comings, we introduce a new large-scale dataset that joins public tree censuses from 23 cities with a large collection of street level and aerial imagery. Our Auto Arborist dataset contains over 2.5M trees and 344 genera and is >2 or-ders of magnitude larger than the closest dataset in the literature. We introduce baseline results on our dataset across modalities as well as metrics for the detailed analy-sis of generalization with respect to geographic distribution shifts, vital for such a system to be deployed at-scale.
BibTeX
@inproceedings{beery2022auto,
  author = {Sara Beery and Guanhang Wu and Trevor Edwards and Filip Pavetic and Bo Majewski and Shreyasee Mukherjee and Stanley Chan and John Morgan and Vivek Rathod and Jonathan Huang},
  title = {{The Auto Arborist Dataset: A Large-Scale Benchmark for Multiview Urban Forest Monitoring Under Domain Shift}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.02061},
}
Figure