FlameFinder: Illuminating Obscured Fire Through Smoke With Attentive Deep Metric Learning

Hossein Rajoli, Sahand Khoshdel, Fatemeh Afghah, Xiaolong Ma

TL;DR

FlameFinder is a novel deep metric learning (DML) framework designed to detect wildfires through thick smoke using UAV-mounted thermal cameras. By employing an attention-weighted combination of triplet and center losses, the system learns to distinguish the structural "signature" of a flame from non-fire hot spots, significantly reducing false positives compared to standard deep learning models.

What Problem Does This Paper Address?

Early and accurate wildfire detection is critical for disaster management, and drones (UAVs) are ideal tools for the job. However, they face a "modality" dilemma:

  1. RGB (Visible Light) Cameras: These provide high-resolution details but are completely blinded by smoke and water vapor.
  2. Thermal Cameras: These can "see" through smoke, but they lack absolute temperature references. Because they rely on relative thermal gradients, they often trigger false positives on "hot spots" that aren't actually fire—such as sun-heated rocks, warm smoke plumes, or reflective surfaces.

The challenge is creating a model that can look at a thermal image and recognize the features of a flame (its shape, distribution, and texture) rather than just looking for the brightest pixels.

Methods

The authors propose a framework that uses Deep Metric Learning (DML). Instead of a simple "fire vs. no-fire" classifier, DML trains the model to map images into an "embedding space" where similar objects are clustered tightly together and dissimilar objects are pushed far apart.

Key technical components include:

  • Multi-Modal Training, Uni-Modal Inference: The model is trained using paired RGB and thermal images. An automated algorithm (FlameSeg) uses the clear RGB images to generate labels, which are then used to "teach" the thermal model what a flame looks like. During actual deployment (inference), the model uses only thermal data to see through smoke.
  • Attentive DML: Standard DML often suffers from "loss domination," where one objective (like Center Loss) overwhelms others. The authors introduced an attention mechanism that dynamically weights three different losses:
  • Center Loss (CL): Minimizes the distance between samples and their class center.
  • Triplet Center Loss (TCL): Ensures the distance to the correct class center is smaller than the distance to the wrong one.
  • Triplet Cosine Center Loss (TCCL): Focuses on the angle between feature vectors, which is more effective for high-dimensional data.
  • Local Feature Extraction (LFE): The architecture uses a U-Net-based encoder to capture both global context and local intricate patterns of flame spots.

Datasets

The study utilizes two primary datasets:

  • FLAME2: A large-scale dataset containing 53,000 paired RGB/Thermal frames captured by drones during prescribed burns in Arizona.
  • FLAME3: A newer dataset (specifically the "Shoetank" subset) used to test the model's ability to generalize to different fire types and environments.

Key Insights and Ideas

  • Beyond Intensity: Most thermal detectors are essentially "brightness detectors." FlameFinder’s use of Cosine and Triplet losses forces the model to learn the geometric and textural relationships of fire, making it much harder to fool with a hot rock or warm smoke.
  • The "Attention" Balancer: The most interesting finding is that the attention mechanism acts as a regularizer for the learning process itself. By preventing Center Loss from dominating, the model is forced to explore non-magnitude features, leading to much better class separation in the latent space.
  • Knowledge Transfer: The paper demonstrates a successful "indirect supervision" strategy. By using heuristic-based RGB segmentation to label thermal data, they bypass the need for massive, manually-labeled thermal datasets, which are notoriously difficult for humans to annotate accurately.

Key Results

FlameFinder consistently outperformed established backbones like ResNet18 and VGG19:

  • Accuracy: Achieved 97.8% on FLAME2 and 88.9% on FLAME3, representing a 4.4% to 7% improvement over baseline models.
  • False Negatives: The model showed a significantly lower False Negative Rate (11.6% vs. 17% for ResNet18), which is vital for early fire detection where a missed flame can lead to a catastrophe.
  • Latent Space Clarity: t-SNE visualizations (dimensionality reduction) showed that FlameFinder creates much tighter, more distinct clusters for "flame" vs. "no-flame" than standard classification approaches.

Limitations and Future Work

  • Label Noise: The automated RGB segmentation (FlameSeg) agreed with human experts about 81% of the time. This means the model is being trained on "noisy" data, which likely caps its maximum potential accuracy.
  • Opaque Obstructions: While thermal cameras see through smoke, they cannot see through solid objects like thick tree canopies or ridges.
  • Future Directions: The authors suggest incorporating temporal data (video sequences) rather than just static frames. Analyzing how a hot spot flickers or moves over time could further distinguish real flames from static hot objects.

Summary updated

Abstract

FlameFinder, a novel deep metric learning (DML) framework, accurately detects RGB-obscured flames using thermal images from firefighter drones during wildfire monitoring. In contrast to RGB, thermal cameras can capture smoke-obscured flame features but they lack absolute thermal reference points, detecting many nonflame hot spots as false positives. This issue suggests that extracting features from both modalities in unobscured cases can reduce the model’s bias to relative thermal gradients. Following this idea, our proposed model utilizes paired thermal-RGB images captured onboard drones for training, learning latent flame features from smoke-free samples. In testing, it identifies flames in smoky patches based on their equivalent thermal-domain distribution, improving performance with supervised and distance-based clustering metrics. The approach includes a flame segmentation method and a DML-aided detection framework with center loss (CL), triplet CL (TCL), and triplet cosine CL (TCCL), to find the optimal cluster representatives for classification. Evaluation of FLAME2 and FLAME3 datasets shows the method’s effectiveness in diverse fire and no-fire scenarios. However, the CL dominates the two other losses, resulting in the model missing features that are sensitive to them. To overcome this issue, an attention mechanism is proposed making nonuniform feature contribution possible and amplifying the critical role of cosine and triplet loss in the DML framework. Plus, the attentive DML shows improved interpretability, class discrimination, and decreased intraclass variance exploiting several other flame-related features. The proposed model surpasses the baseline with a binary classifier by 4.4% in FLAME2 and 7% in FLAME3 datasets for unobscured flame detection accuracy. It also demonstrates enhanced class separation in obscured scenarios compared to fine-tuned VGG19, ResNet18, and three other backbone models tailored for flame detection.
BibTeX
@article{rajoli2024flamefinder,
  author = {Hossein Rajoli and Sahand Khoshdel and Fatemeh Afghah and Xiaolong Ma},
  title = {{FlameFinder: Illuminating Obscured Fire Through Smoke With Attentive Deep Metric Learning}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2024},
  doi = {10.1109/tgrs.2024.3440880},
}

Loading related content...

Figure