Introduction
Wildfire detection is a high-stakes "needle in a haystack" problem. Smoke plumes are spatially and temporally rare, meaning training data is often scarce and imbalanced. This leads to two major issues: overfitting, where models rely on specific background features rather than the smoke itself, and ambiguity, where smoke looks remarkably similar to clouds, fog, or camera glare, leading to high false-positive rates.
While Deep Neural Networks (DNNs) have shown promise in detecting incipient fires, their robustness in real-world conditions remains poorly understood. Environmental factors such as dirty camera lenses or drifting clouds can significantly impact system reliability. This research introduces WARP (Wildfire Adversarial Robustness Procedure), the first model-agnostic framework designed to stress-test wildfire detection models against environmental noise and deceptive visual artifacts.
The WARP Framework
The proposed WARP framework evaluates model resilience through two distinct types of adversarial "attacks" that simulate real-world challenges:
- Global Sanity Check (Noise Overlay): This simulates image degradation caused by network latency or sensor noise by adding Gaussian noise across the entire image. Performance is measured using the mAP Percentage Loss ($L$) to quantify how quickly precision degrades as noise levels increase.
- Local Deception Test (Noise Patch): This targets the model's ability to distinguish smoke from look-alikes. Small PNG patches of clouds are injected into a $25 \times 25$ grid across the image. The framework tracks Classification Flip Probabilities—how often a "smoke" image becomes "null" ($\alpha$) or a "null" image is falsely flagged as "smoke" ($\beta$)—and the Localization Deception Rate ($\gamma$), which measures how often the model incorrectly draws a bounding box around a cloud patch.
Crucially, WARP is model-agnostic, requiring no access to internal gradients, which allows it to be applied to any new architecture, from traditional Convolutional Neural Networks (CNNs) to modern Transformers.
Experimental Setup and Datasets
The study utilized two primary data sources to evaluate state-of-the-art models (YOLOv8 and RT-DETR):
- NEMO (NEvada sMOke detection benchmark): Used for training and validation, containing over 2,500 images of cropped and zoomed smoke-positive regions.
- HPWREN (High-Performance Wireless Research and Education Network): Used for testing and robustness evaluation. This dataset contains 1,661 unedited, realistic smoke-positive images, providing a challenging real-world testbed.
Key Insights and Results
The evaluation revealed significant trade-offs between different model architectures:
- Architectural Vulnerabilities: While Transformers (RT-DETR) often achieve higher precision on clean data, they proved to be significantly more "brittle." Under global noise, the Transformer's mAP loss ($L$) reached 86.53% at a noise level of $\alpha = 0.210$, compared to only 50.82% for the CNN (YOLOv8).
- The "Cloud" Confusion: Both models struggled with local cloud patches. The probability of a smoke-positive image being "flipped" to negative by a single cloud patch was roughly 50% for CNNs and 34% for Transformers. However, the Localization Deception Rate for the Transformer was more than 3x higher than the CNN, indicating that Transformers are much more prone to "hallucinating" objects within noisy patches.
- Annotation Bias: A notable "center-bias" was discovered. Because training data typically features smoke near the horizon (the middle-horizontal area), models became hyper-focused on that region, making them more susceptible to deceptions placed there.
Conclusion and Future Work
The vulnerabilities identified by WARP suggest that current wildfire detection models require more than just high baseline precision; they require targeted robustness. The authors propose using WARP’s noise and patch injection techniques as data augmentation strategies during training to "teach" models to distinguish between smoke and environmental artifacts. Future work will focus on a "closing the loop" study to measure how much these wildfire-oriented augmentations improve robustness scores in subsequent testing rounds.
Summary updated