Abstract
Accurate detection of individual tree crowns from remote sensing data poses a significant challenge due to the dense nature of forest canopy and the presence of diverse environmental variations, e.g., overlapping canopies, occlusions, and varying lighting conditions. Additionally, the lack of data for training robust models adds another limitation in effectively studying complex forest conditions. This paper presents a novel method for detecting shadowed tree crowns and provides a challenging dataset comprising roughly 50k paired RGB-thermal images to facilitate future research for illumination-invariant detection. The proposed method (ShadowSense) is entirely self-supervised, leveraging domain adversarial training without source domain annotations for feature extraction and foreground feature alignment for feature pyramid networks to adapt domain-invariant representations by focusing on visible foreground regions, respectively. It then fuses complementary information of both modalities to effectively improve upon the predictions of an RGB-trained detector and boost the overall accuracy. Extensive experiments demonstrate the superiority of the proposed method over both the baseline RGB-trained detector and state-of-the-art techniques that rely on unsupervised domain adaptation or early image fusion.
Introduction
Individual Tree Crown Detection (ITCD) is a cornerstone of modern forestry, essential for monitoring forest health, estimating carbon storage, and managing pest infestations. While drone-based RGB cameras provide high-resolution data, they have a major blind spot: shadows. In dense canopies, smaller or lower-tier trees are often completely obscured by the shadows of their taller neighbors. Thermal sensors can "see" these hidden trees because thermal signatures are less dependent on visible illumination. However, two major hurdles exist: the lack of large-scale, high-quality datasets containing paired RGB and thermal drone imagery, and the "annotation bottleneck" where labeling thermal images is difficult and time-consuming.
To address these challenges, we introduce RT-Trees, a significant new benchmark featuring roughly 50,000 paired and co-registered RGB-thermal images captured over central Canadian forests. Unlike previous datasets focused on urban areas, RT-Trees is specifically tailored for complex forest environments.
Methodology
We propose ShadowSense, a framework designed to adapt a model trained on well-lit RGB images to work effectively with thermal data using Unsupervised Domain Adaptation (UDA). This approach bypasses the need for manual thermal labels by treating registered RGB-thermal pairs as their own source of supervision.
The architecture incorporates three key components:
- Domain Adversarial Training (DAT): The model employs a discriminator to distinguish between RGB and thermal feature maps. The feature extractor is trained to "fool" this discriminator, forcing it to learn domain-invariant features.
- Foreground FPN Feature Alignment: To ensure the model focuses on relevant objects, a classical Watershed segmentation algorithm creates binary masks of visible trees. Feature maps from the RGB and thermal branches are then forced to align specifically within these "foreground" regions. This prevents the model from being distracted by forest backgrounds (ground, debris) which look fundamentally different across modalities.
- Masked Feature Fusion: During inference, the system performs a weighted average of features. It prioritizes high-detail RGB for well-lit areas and blends in thermal features for shadowed "background" areas to reveal hidden tree crowns.
Results and Discussion
Extensive experiments demonstrate that ShadowSense significantly outperforms standard RGB detectors and state-of-the-art UDA methods.
- Overall Accuracy: ShadowSense achieved 54.13% AP50, a substantial improvement over the baseline RGB-only model (49.86%).
- Shadow Performance: In the "difficult" (shadowed) category, ShadowSense correctly identified 19.09% of trees, nearly doubling the 10.41% success rate of the standard RGB-only detector.
- Comparison to UDA: ShadowSense also outperformed general-purpose UDA methods like DA-RetinaNet, proving that foreground-specific alignment is more effective for forestry applications than global alignment strategies.
Conclusion
ShadowSense represents a major step forward in shadow-agnostic tree crown detection. By leveraging self-supervision and foreground-aware feature alignment, it effectively utilizes thermal imagery to "see through" forest shadows without requiring expensive manual annotations. While current limitations include a dependency on precise image registration and the lower resolution of thermal sensors, the framework provides a robust foundation for future research in multi-modal remote sensing.
Source: ShadowSense: Unsupervised Domain Adaptation and Feature Fusion for Shadow-Agnostic Tree Crown Detection from RGB-Thermal Drone Imagery (WACV 2024).
Summary updated