Assessing SAM for Tree Crown Instance Segmentation from Drone Imagery

Mélisande Teng, Arthur Ouaknine, Etienne Laliberté, Yoshua Bengio, David Rolnick, Hugo LarochellearXiv 2025 · OpenReview

TL;DR

This paper investigates the effectiveness of the Segment Anything Model (SAM) for tree crown instance segmentation in drone imagery, a crucial task for monitoring reforestation efforts. They found that SAM, out-of-the-box, doesn't outperform a custom Mask R-CNN, but that task-specific adaptations and the inclusion of Digital Surface Model (DSM) data significantly improve performance, suggesting SAM's potential when fine-tuned.

What Problem Does This Paper Address?

Accurately monitoring tree planting projects is vital for assessing their effectiveness as climate solutions. Traditional manual surveys are labor-intensive and costly. While drone imagery and deep learning offer a promising automated alternative, precisely segmenting individual tree crowns, especially in young plantations where crowns are less distinct, remains challenging due to a lack of large-scale, precisely annotated datasets. This paper explores whether a powerful general-purpose segmentation model like SAM can be effectively adapted for this specific, high-resolution remote sensing task.

Methods

The researchers evaluated several approaches using SAM for tree crown instance segmentation:

  • SAM Out-of-the-Box: They tested SAM in its automatic mask generation mode and with point prompts derived from local elevation maxima in the Digital Surface Model (DSM).
  • SAM with Prompts from Other Models: They explored using bounding boxes and masks generated by a pre-trained Mask R-CNN as prompts for SAM.
  • Task-Specific SAM Adaptation: They investigated methods that fine-tune SAM components, specifically building upon the RSPrompter approach. This involved creating "DSMPrompter," which incorporates a trainable DSM encoder to provide richer spatial information to SAM.
  • Baseline Models: For comparison, they used a standard Mask R-CNN and Faster R-CNN, both with and without DSM input.

A key aspect of their work was exploring the utility of Digital Surface Models (DSM), which capture elevation data and can be derived from drone photogrammetry, as an additional input alongside RGB imagery.

Datasets

The study utilized the UAV Canadian (Quebec) Plantations dataset. This dataset consists of high-resolution (5mm/pixel) RGB orthomosaics and corresponding Digital Surface Models (DSMs) from young tree plantations. The dataset includes annotations for tree crown delineations and species labels. The researchers preprocessed this dataset, excluding certain sites and filtering tiles to ensure quality and relevance for the task, resulting in 26,428 tiles for training, validation, and testing.

Key Insights and Ideas

  • SAM's "Out-of-the-Box" Limitations: The paper highlights that SAM, despite its impressive general segmentation capabilities, struggles with precise instance segmentation of tree crowns in high-resolution drone imagery without task-specific adaptation. It often produces false positives or merges touching crowns.
  • The Power of Prompts: While automatic SAM is weak, using well-designed prompts (like local elevation maxima from DSM) significantly improves its performance, though still not surpassing a custom Mask R-CNN.
  • DSM is Crucial: Incorporating DSM data consistently improves the performance of all tested models, including SAM-based approaches and Mask R-CNN. This suggests that 3D structural information is highly valuable for distinguishing tree crowns.
  • Fine-tuning SAM is Key: Methods that adapt or fine-tune SAM components, like DSMPrompter, show significant promise and can achieve competitive or superior results compared to standard models.
  • Artifacts Matter: The quality of the input orthomosaic, potentially containing photogrammetry artifacts, can negatively impact segmentation results, even when using advanced models.

Key Results

  • SAM Out-of-the-Box vs. Mask R-CNN: SAM in automatic mode and with DSM prompts performed significantly worse than a custom Mask R-CNN. For instance, Mask R-CNN achieved mAP scores in the high 60s, while SAM automatic was in the low single digits.
  • DSM Improvement: Adding DSM input improved Mask R-CNN performance by approximately 2-3% in mAP.
  • Prompting SAM with Mask R-CNN: Surprisingly, using Mask R-CNN outputs as prompts for SAM degraded performance compared to using Mask R-CNN alone.
  • RSPrompter and DSMPrompter: The adapted SAM methods, particularly DSMPrompter, achieved the highest performance, with DSMPrompter reaching an mAP of 64.84 (±1.49) and a weighted mAP (wmAP) of 64.84 (±1.49), outperforming other methods.
  • Qualitative Findings: SAM automatic often struggled with separating touching crowns and sometimes segmented background elements. DSMPrompter produced sharper segmentations than RSPrompter.

Limitations and Future Work

  • SAM's "Out-of-the-Box" Performance: The study clearly demonstrates that SAM is not a plug-and-play solution for this specific task and requires significant adaptation or careful prompting.
  • Artifact Sensitivity: The presence of photogrammetry artifacts in the input imagery can degrade segmentation quality, as observed when using Mask R-CNN outputs as SAM prompts.
  • Future Directions: The authors suggest exploring different ways to integrate DSM information, assessing the impact of downsampling input imagery resolution, and further investigating DSMPrompter's potential in low-data regimes. They also note the need for task-specific tuning of SAM for it to be advantageous over simpler custom models.

Summary updated

Abstract

The potential of tree planting as a natural climate solution is often undermined by inadequate monitoring of tree planting projects. Current monitoring methods involve measuring trees by hand for each species, requiring extensive cost, time, and labour. Advances in drone remote sensing and computer vision offer great potential for mapping and characterizing trees from aerial imagery, and large pre-trained vision models, such as the Segment Anything Model (SAM), may be a particularly compelling choice given limited labeled data. In this work, we compare SAM methods for the task of automatic tree crown instance segmentation in high resolution drone imagery of young tree plantations. We explore the potential of SAM for this task, and find that methods using SAM out-of-the-box do not outperform a custom Mask R-CNN, even with well-designed prompts, but that there is potential for methods which tune SAM further. We also show that predictions can be improved by adding Digital Surface Model (DSM) information as an input.
BibTeX
@misc{teng2025assessing,
  author = {Mélisande Teng and Arthur Ouaknine and Etienne Laliberté and Yoshua Bengio and David Rolnick and Hugo Larochelle},
  title = {{Assessing SAM for Tree Crown Instance Segmentation from Drone Imagery}},
  year = {2025},
  eprint = {2503.20199},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.20199},
}
Figure