ShadowSense: Unsupervised Domain Adaptation and Feature Fusion for Shadow-Agnostic Tree Crown Detection from RGB-Thermal Drone Imagery

Rudraksh Kapil, Seyed Mojtaba Marvasti-Zadeh, Nadir Erbilgin, Nilanjan Ray

Abstract

Accurate detection of individual tree crowns from remote sensing data poses a significant challenge due to the dense nature of forest canopy and the presence of diverse environmental variations, e.g., overlapping canopies, occlusions, and varying lighting conditions. Additionally, the lack of data for training robust models adds another limitation in effectively studying complex forest conditions. This paper presents a novel method for detecting shadowed tree crowns and provides a challenging dataset comprising roughly 50k paired RGB-thermal images to facilitate future research for illumination-invariant detection. The proposed method (ShadowSense) is entirely self-supervised, leveraging domain adversarial training without source domain annotations for feature extraction and foreground feature alignment for feature pyramid networks to adapt domain-invariant representations by focusing on visible foreground regions, respectively. It then fuses complementary information of both modalities to effectively improve upon the predictions of an RGB-trained detector and boost the overall accuracy. Extensive experiments demonstrate the superiority of the proposed method over both the baseline RGB-trained detector and state-of-the-art techniques that rely on unsupervised domain adaptation or early image fusion.

Introduction

Individual Tree Crown Detection (ITCD) is a cornerstone of modern forestry, essential for monitoring forest health, estimating carbon storage, and identifying species. However, standard deep learning models rely almost exclusively on RGB (visible light) imagery. In dense forests, this creates a "shadow problem": taller trees cast deep shadows that completely obscure smaller trees beneath them. To an RGB sensor, these areas appear as featureless dark regions. While thermal sensors can detect the heat signatures of these hidden trees regardless of lighting, thermal data lacks high-resolution texture and is notoriously difficult to label manually.

ShadowSense addresses this gap by effectively fusing high-detail RGB and illumination-invariant thermal modalities without requiring a massive, manually labeled thermal dataset. By treating the spatial alignment of the two cameras as a form of free supervision, the framework nearly doubles the detection rate of shadowed trees compared to standard RGB-only models.

Methods

The authors propose a dual-branch architecture based on the RetinaNet detector. The entire adaptation process is self-supervised, leveraging the existing weights of an RGB-trained detector (DeepForest) and the spatial alignment of the two sensors.

The complete training workflow of ShadowSense, highlighting the dual-branch architecture, domain adversarial training, and foreground feature alignment.
  1. Domain Adversarial Training (DAT): To make thermal data compatible with an RGB-trained model, a discriminator network is used. It attempts to distinguish between RGB and thermal feature maps, while the thermal branch is trained to "fool" it, forcing the extraction of domain-invariant features.
  2. Foreground (FG) Feature Alignment: Using a classic watershed segmentation algorithm on the RGB image, the system creates a binary mask of foreground (trees) vs. background (shadows). Thermal features are then forced to align with RGB features specifically in these foreground regions using an L2 loss.
  3. Masked Feature Fusion: During inference, the model performs fusion at the Feature Pyramid Network (FPN) level. It uses the generated mask to preserve high-quality RGB features for well-lit areas while "filling in" shadowed areas with features extracted from the thermal branch.

RT-Trees Dataset

The paper introduces RT-Trees, a significant new benchmark for the community:

  • Scale: Contains 49,879 paired RGB-thermal image pairs captured via drone over Canadian forests, a massive leap over previous datasets like TNO or MFNet.
  • Evaluation: While the training set is unlabeled, the authors manually annotated a subset of 63 images (containing ~3,800 trees) to rigorously test the model's ability to find difficult, shadowed crowns.

Key Results

ShadowSense significantly outperformed both the baseline and state-of-the-art (SOTA) unsupervised domain adaptation methods across several metrics:

Qualitative comparison of detection results showing ShadowSense's superior performance in shadowed regions.
  • Overall Accuracy: Achieved an AP50 of 54.13%, compared to 49.86% for the off-the-shelf RGB baseline.
  • Shadowed Tree Detection: ShadowSense correctly identified 19.09% of shadowed trees, nearly doubling the baseline's performance of 10.41%.
  • Robustness: It outperformed popular methods like PearlGAN (thermal colorization) and UMFusion, which often introduced artifacts that degraded detection performance.

Conclusion and Future Work

ShadowSense demonstrates that classic computer vision techniques like watershed segmentation can effectively guide deep learning models in multi-modal tasks. By fusing features at the FPN level rather than the image level, the model avoids common artifacts and successfully leverages the complementary strengths of RGB and thermal sensors.

Future research directions include exploring more sophisticated attention-based fusion mechanisms and testing the framework on other multi-modal tasks where one sensor is periodically blinded, such as autonomous driving in fog or night conditions.

Summary updated

Abstract

Accurate detection of individual tree crowns from remote sensing data poses a significant challenge due to the dense nature of forest canopy and the presence of diverse environmental variations, e.g., overlapping canopies, occlusions, and varying lighting conditions. Additionally, the lack of data for training robust models adds another limitation in effectively studying complex forest conditions. This paper presents a novel method for detecting shadowed tree crowns and provides a challenging dataset comprising roughly 50k paired RGB-thermal images to facilitate future research for illumination-invariant detection. The proposed method (ShadowSense) is entirely self-supervised, leveraging domain adversarial training without source domain annotations for feature extraction and foreground feature alignment for feature pyramid networks to adapt domain-invariant representations by focusing on visible foreground regions, respectively. It then fuses complementary information of both modalities to effectively improve upon the predictions of an RGB-trained detector and boost the overall accuracy. Extensive experiments demonstrate the superiority of the proposed method over both the baseline RGB-trained detector and state-of-the-art techniques that rely on unsupervised domain adaptation or early image fusion. Our code and data are available: https://github.com/rudrakshkapil/ShadowSense
BibTeX
@inproceedings{kapil2024shadowsense,
  author = {Rudraksh Kapil and Seyed Mojtaba Marvasti-Zadeh and Nadir Erbilgin and Nilanjan Ray},
  title = {{ShadowSense: Unsupervised Domain Adaptation and Feature Fusion for Shadow-Agnostic Tree Crown Detection from RGB-Thermal Drone Imagery}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=NLF2uCp0vf},
}
Figure