TL;DR
This paper tackles the challenge of real-time wildfire smoke detection from UAVs, where onboard computational power is limited. They propose a novel approach using "knowledge distillation" to train efficient segmentation models by leveraging large, pre-trained "teacher" models (like SAM) to generate pseudo-labels from simple bounding box annotations. This method significantly reduces the need for extensive pixel-level annotated data, enabling accurate smoke segmentation on resource-constrained UAVs.
What Problem Does This Paper Address?
Early detection of wildfires is crucial to mitigate their devastating environmental and societal impacts. Uncrewed Aerial Vehicles (UAVs) offer a promising platform for rapid, wide-area surveillance, especially in remote areas. However, the limited computational power and lack of high-bandwidth connectivity on these UAVs pose a significant challenge for real-time, accurate wildfire detection. Specifically, while bounding box detection methods are computationally lighter, they lack the precision needed for accurate localization. Pixel-level segmentation offers better localization but typically requires vast amounts of meticulously annotated data, which is scarce for wildfire smoke. This paper aims to bridge this gap by enabling accurate smoke segmentation on resource-constrained UAVs, even with limited pixel-level training data.
Methods
The core of the proposed method is knowledge distillation, a technique where a smaller "student" model learns from a larger, more capable "teacher" model. Here, the student model is a lightweight segmentation network called PIDNet, chosen for its real-time inference capabilities.
The key innovation lies in how the pseudo-labels for training the student model are generated:
- Bounding Box Guided Pseudo-Labeling: Instead of requiring full pixel-level masks, the authors leverage existing bounding box annotations.
- Teacher Models: Two types of teacher models are explored:
- Segment Anything Model (SAM): A powerful, prompt-guided zero-shot segmentation model. SAM is used directly with the bounding box annotations as prompts to generate initial pseudo-labels.
- Larger Segmentation Models: These are trained using state-of-the-art bounding box supervision methods (like BoxSnake) and then used to infer segmentation masks for the entire dataset.
- Distillation Loss: The PIDNet student model is trained to minimize the difference between its own predictions and the pseudo-labels generated by the teacher model. This is achieved through a combination of loss functions, including binary cross-entropy for segmentation outputs and losses that incorporate boundary information.
This approach allows the student model to learn complex segmentation patterns from the teacher without needing direct pixel-level supervision on the training data.
Datasets
The study utilizes a combination of datasets for training and evaluation:
- AI For Mankind and HPWREN Wildfire Smoke Dataset (Versions 1.0 and 2.0): This dataset provides bounding box annotations for wildfire smoke.
- UAV-collected and manually bounding box labelled data (Raita-Hakola et al.): This dataset consists of 5035 bounding box annotated UAV-captured aerial footage of boreal forests, with a high percentage containing visible smoke.
- Smokeless Images: A set of 623 smokeless images from AI For Mankind and HPWREN were used to evaluate false positive rates.
- Croatian Center for Wildfire Research Dataset: A smaller, human-labelled test set of 49 images with segmentation labels ("smoke" and "maybe smoke") was used for generalization testing.
The authors created a three-part split (training, validation, testing) for the primary datasets, ensuring temporal separation for UAV data and clustering-based separation for the AI For Mankind data to avoid major similarities between sets.
Key Insights and Ideas
- Bounding Boxes are Enough (with help): The study demonstrates that even simple bounding box annotations can be effectively leveraged to train sophisticated pixel-level segmentation models, significantly reducing annotation effort.
- Foundation Models as Teachers: Large foundation models like SAM can act as powerful "teachers" to distill knowledge into smaller, real-time capable models, even without task-specific fine-tuning of the teacher itself.
- Real-time Segmentation on Edge Devices: The research successfully deploys a trained PIDNet model on an NVIDIA Jetson Orin NX, achieving real-time inference speeds (~25 fps) suitable for onboard UAV applications.
- Generalization to Diverse Conditions: The trained model showed promising generalization capabilities, performing well on unseen data from different geographical locations and under challenging lighting and contrast conditions.
Key Results
- Accuracy: The proposed method achieved a mean Intersection over Union (mIoU) of 63.3% on a manually annotated and diverse wildfire dataset.
- Real-time Performance: The PIDNet-S model achieved approximately 25 fps inference speed on the NVIDIA Jetson Orin NX, making it suitable for onboard processing.
- Detection Range: The system successfully detected smoke from distances up to 9.7 km in real-world forest burning events.
- Comparison: The distillation approach using SAM or other teacher models generally outperformed direct bounding box supervision methods.
Limitations and Future Work
- Data Distribution: The study acknowledges that the greatest limitation is the narrow geographical and visual distribution of the training data. Future work should involve extending this distribution with more diverse weak supervision methods or diffusion model-generated data.
- False Positives: The models exhibited a tendency to generate false positives, particularly in regions with clouds or fog. Future work suggests combating this by including more diverse smokeless images in training and potentially optimizing the loss terms.
- Pseudo-label Quality: The quality of SAM-generated pseudo-labels was not perfect, with visibly misplaced boundaries in some cases. Future work could explore improving pseudo-label generation through prompt variations or fine-tuning SAM specifically for smoke segmentation.
- Loss Function Optimization: The optimal setting for the boundary-dependent loss terms was not fully explored, suggesting further optimization is needed.
- Comparison to Fully Supervised Methods: Due to the lack of densely labelled data, a direct comparison to fully supervised methods was not performed, which is a potential area for future research.
Summary updated