RT-Trees: Evaluation and RGB training images with masks

Rudraksh Kapil, Seyed Mojtaba Marvasti-Zadeh, Nadir Erbilgin, Nilanjan Ray

TL;DR

ShadowSense is a new self-supervised framework that enables drone-based tree detection to "see" through deep shadows by fusing RGB and thermal imagery. By leveraging a novel foreground-focused feature alignment and a massive new dataset of 50,000 image pairs, the system significantly improves detection accuracy in dense forests without requiring any manual thermal-image annotations.

What Problem Does This Paper Address?

Individual Tree Crown Detection (ITCD) is a cornerstone of modern forestry, essential for everything from carbon storage estimation to monitoring pest infestations. While deep learning models like DeepForest have become quite good at identifying trees in clear, well-lit RGB drone imagery, they have a major blind spot: shadows.

In dense forest canopies, taller trees cast deep shadows that completely obscure smaller or neighboring trees in the visible spectrum (RGB). Thermal sensors can "see" these hidden trees because thermal signatures are less dependent on visible illumination. However, we lack large-scale, annotated thermal datasets to train these models. Furthermore, simply "stitching" RGB and thermal data together often leads to artifacts or a loss of detail. The challenge is: how do we teach a model to use thermal data to find shadowed trees when we don't have labels for those thermal images?

Methods

The authors introduce ShadowSense, a dual-branch architecture based on RetinaNet that uses a "teacher-student" style approach to adapt to thermal data without new labels.

  1. Domain Adversarial Training (DAT): The model features two parallel branches—one for RGB (frozen with pre-trained weights) and one for Thermal. A "discriminator" tries to guess if a feature map came from the RGB or Thermal branch. The Thermal branch is trained to "fool" this discriminator, forcing it to learn features that are "domain-invariant" (meaning they look the same regardless of the sensor type).
  2. Foreground (FG) Feature Alignment: Standard domain adaptation often fails because it tries to align the entire image (including irrelevant ground or sky). ShadowSense uses a simple, classic watershed segmentation algorithm to create "foreground masks" of trees. It then forces the RGB and Thermal feature maps to match specifically in these tree-filled regions using a Mean Squared Error (MSE) loss.
  3. Masked Feature Fusion: During inference, the model doesn't just pick one sensor. It uses the generated mask to identify background/shadowed areas. In well-lit areas, it relies on high-detail RGB. In shadowed areas, it performs a weighted average of the RGB and Thermal features, effectively "filling in" the visual gaps with thermal data.

Datasets

The paper introduces RT-Trees, a significant contribution to the field.

  • Scale: Roughly 50,000 paired RGB-thermal image pairs.
  • Diversity: Data was collected across 14 drone flights in central Canada, capturing various lighting conditions, temperatures, and snow cover.
  • Comparison: Most existing RGB-Thermal datasets (like TNO or MFNet) have fewer than 2,000 pairs. RT-Trees is an order of magnitude larger, specifically tailored for forest environments.
  • Evaluation: While the training set is unlabeled, the authors manually annotated 63 non-overlapping images (containing ~3,700 trees) to create a rigorous benchmark for testing.

Key Insights and Ideas

  • The "Aha" of Foreground Focus: The authors found that aligning entire feature maps actually hurt performance. By using a "dumb" but effective watershed mask to focus the model's attention on the trees (the foreground), the model learned much more relevant features.
  • Self-Supervision is Key: By using a pre-trained RGB detector as a "source of truth," the model can adapt to the thermal domain without a human ever having to draw a box on a thermal image.
  • Shadow-Agnosticism: The system doesn't just detect trees better; it specifically targets the "difficult" cases. It treats shadows not as missing data, but as a signal to lean more heavily on the thermal branch.

Key Results

ShadowSense was tested against standard RGB detectors and several state-of-the-art unsupervised domain adaptation (UDA) methods:

  • Overall Accuracy: ShadowSense achieved an AP50 of 54.13%, outperforming the baseline RGB-only detector (49.86%).
  • Shadow Detection: The most striking result is in the "Shadowed Trees" category. The baseline model only identified 10.41% of trees hidden in shadows. ShadowSense nearly doubled this, identifying 19.09%.
  • Comparison to SOTA: It outperformed other UDA methods (like DA-RetinaNet) and image-to-image translation methods (like PearlGAN), which often introduced artifacts that confused the detector.

Limitations and Future Work

While ShadowSense is a leap forward, the authors note a few areas for improvement:

  • The Watershed Bottleneck: The foreground mask generation relies on the assumption that trees are brighter than the background in RGB. While generally true, this might fail in very complex lighting or with specific soil types.
  • Thermal Resolution: Thermal images have lower spatial resolution than RGB. Even with fusion, the model sometimes struggles with the fine-grained boundaries of very small shadowed trees.
  • Future Directions: The authors suggest exploring more sophisticated "attention-based" fusion modules and testing the model's robustness across different forest types (e.g., tropical vs. boreal) where thermal signatures might differ significantly.

Summary updated

Abstract

Accurate detection of individual tree crowns from remote sensing data poses a significant challenge due to the dense nature of forest canopy and the presence of diverse environmental variations, e.g., overlapping canopies, occlusions, and varying lighting conditions. Additionally, the lack of data for training robust models adds another limitation in effectively studying complex forest conditions. This paper presents a novel method for detecting shadowed tree crowns and provides a challenging dataset comprising roughly 50k paired RGB-thermal images to facilitate future research for illumination-invariant detection. The proposed method (ShadowSense) is entirely self-supervised, leveraging domain adversarial training without source domain annotations for feature extraction and foreground feature alignment for feature pyramid networks to adapt domain-invariant representations by focusing on visible foreground regions, respectively. It then fuses complementary information of both modalities to effectively improve upon the predictions of an RGB-trained detector and boost the overall accuracy. Extensive experiments demonstrate the superiority of the proposed method over both the baseline RGB-trained detector and state-of-the-art techniques that rely on unsupervised domain adaptation or early image fusion. Our code and data are available: https://github.com/rudrakshkapil/ShadowSense
BibTeX
@inproceedings{kapil2024rt,
  author = {Rudraksh Kapil and Seyed Mojtaba Marvasti-Zadeh and Nadir Erbilgin and Nilanjan Ray},
  title = {{RT-Trees: Evaluation and RGB training images with masks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=NLF2uCp0vf},
}
Figure