AusSmoke meets MultiNatSmoke: a fully-labelled diverse smoke segmentation dataset

Weihao Li, Hongjin Zhao, Gao Zhu, Ge-Peng Ji, Nicholas Wilson, Marta Yebra, Nick Barnes

TL;DR

Wildfire smoke detection is crucial, but existing datasets are limited in scale and geographic diversity, hindering model generalization. This paper introduces AusSmoke, a new Australian wildfire smoke dataset, and MultiNatSmoke, a significantly larger, globally diverse benchmark. These resources enable more robust smoke segmentation models, leading to improved early detection capabilities.

What Problem Does This Paper Address?

Wildfires pose a growing threat worldwide, causing devastating environmental, economic, and human impacts. Early detection of wildfires is critical for effective response and mitigation. Camera-based smoke detection, powered by AI, offers a promising solution for rapid identification. However, the development of accurate and generalizable AI models for smoke segmentation is hampered by a lack of comprehensive datasets. Existing datasets often suffer from:

  • Limited Scale: The largest publicly available real-image datasets contain only a few thousand images.
  • Geographic Bias: Many datasets are collected from a single region (e.g., the United States), failing to capture the diverse visual characteristics of smoke under different environmental conditions.
  • Reliance on Synthetic Data: A significant portion of existing datasets uses synthetic images, which can create a "domain gap" between training and real-world scenarios, leading to poor performance in practical applications.

These limitations hinder the training and evaluation of AI models, preventing them from effectively generalizing to diverse wildfire scenarios.

Methods

The paper introduces two key contributions:

  1. AusSmoke Dataset Creation: The authors collected a new dataset of over 15,000 real-world wildfire smoke images specifically from Australia. This dataset emphasizes early-stage fires and includes images from planned burns, which are valuable for capturing the initial signs of smoke under controlled conditions. Images were captured using both fixed PTZ cameras on communication towers and handheld video cameras.
  2. MultiNatSmoke Benchmark Construction: To address the scale and geographic diversity limitations, the authors created a large-scale benchmark called MultiNatSmoke. This benchmark consolidates publicly available international wildfire smoke datasets with their newly collected AusSmoke data. The goal is to create a significantly larger and more geographically diverse dataset than previously available, comprising over 70,000 real smoke images.

The paper also details a multi-stage annotation process to ensure high-quality, pixel-level segmentation masks. This involved manual verification of existing annotations, using the Segment Anything Model (SAM) for initial mask generation from bounding boxes, and a self-training strategy to refine segmentation performance.

Datasets

  • AusSmoke: A newly collected dataset of over 15,000 real-world wildfire smoke images from Australia, emphasizing early-stage fires and planned burns.
  • MultiNatSmoke: A large-scale benchmark (over 70,000 real images) created by combining AusSmoke with several publicly available international datasets. These include:
  • United States: FIgLib, SmokeSeg, Smoke5K
    • Finland: Boreal-Forest-Fire
    • Thailand: FireSpot
    • Brazil: D-Fire
    • Croatia: FESB-MLID
    • China: Forest Fire
    • Vietnam: WSDataset

Key Insights and Ideas

  • Geographic Diversity is Crucial: The study strongly suggests that training models on geographically diverse datasets significantly improves their generalization capabilities compared to models trained on data from a single region. This is a critical takeaway for building robust AI systems for real-world applications.
  • Early Smoke Detection is Challenging: The dataset analysis reveals that a large proportion of smoke images in AusSmoke and MultiNatSmoke contain small smoke regions. This highlights the inherent difficulty in detecting early-stage fires, which are often faint and blend into the background.
  • Synthetic Data Limitations: The paper reinforces the idea that while synthetic data can be useful, it often fails to capture the full complexity of real-world smoke, and models trained solely on synthetic data may not perform well in practice.
  • Scale Matters: The experiments demonstrate a clear trend: increasing the size of the training dataset leads to improved performance across various smoke segmentation models.

Key Results

  • SegFormer Dominates: The SegFormer model consistently achieved the best performance across various metrics (IoU, F1, MSE, Precision) on the MultiNatSmoke test set, outperforming CNN-based and other transformer-based architectures.
  • FoSp Excels in Recall: While SegFormer led overall, the FoSp model achieved the highest recall, indicating its strength in detecting as many smoke pixels as possible, which is crucial for early detection.
  • MultiNatSmoke Outperforms: Models trained on the MultiNatSmoke benchmark consistently outperformed those trained on smaller, less diverse datasets like SmokeSeg and Smoke5K, especially for small smoke regions.
  • Geo-Diversity Boosts Performance: Evaluating SegFormer on the Boreal and AusSmoke test sets showed that training on the geographically diverse MultiNatSmoke subset led to significantly higher IoU and F1 scores compared to training on the SmokeSeg dataset.

Limitations and Future Work

  • Small Smoke Remains a Challenge: Despite improvements, the paper notes that even the best-performing models achieve an IoU below 74% for general smoke segmentation, and performance drops below 62% for small smoke regions. This indicates that accurately segmenting faint, early-stage smoke remains a significant challenge.
  • Further Model Development: The authors suggest that future work could focus on developing more specialized architectures or training strategies to better handle the challenges of small and diffuse smoke.
  • Real-time Performance: While not explicitly detailed as a limitation, the focus on segmentation accuracy implies that further research might be needed to optimize these models for real-time deployment on edge devices for immediate alerts.
  • Data Augmentation and Synthesis: Exploring more advanced data augmentation techniques or novel synthetic data generation methods that better bridge the domain gap could be a future direction.

Summary updated

Abstract

Wildfires are an escalating global concern due to the devastating impacts on the environment, economy, and human health, with notable incidents such as the 2019-2020 Australian bushfires and the 2025 California wildfires underscoring the severity of these events. AI-enabled camera-based smoke detection has emerged as a promising approach for the rapid detection of wildfires. However, existing wildfire smoke segmentation datasets that are used for training detection and segmentation models are limited in scale, geographically constrained, and often rely on synthetic imagery, which hinders effective training and generalization. To overcome these limitations, we present AusSmoke, a new smoke segmentation dataset collected from Australia to address the data scarcity in this region. Furthermore, we introduce a MultiNational geographically diverse and substantially larger fully-labelled benchmark, called MultiNatSmoke, that consolidates publicly available international datasets with the newly collected Australian imagery, expanding the scale by an order of magnitude over previous collections. Finally, we benchmark smoke segmentation models, demonstrating improved performance and enhanced generalization across diverse geographical contexts. The project is available at \href{https://github.com/henryzhao0615/MultiNatSmoke}{Github}.
BibTeX
@inproceedings{li2026aussmoke,
  author = {Weihao Li and Hongjin Zhao and Gao Zhu and Ge-Peng Ji and Nicholas Wilson and Marta Yebra and Nick Barnes},
  title = {{AusSmoke meets MultiNatSmoke: a fully-labelled diverse smoke segmentation dataset}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2026},
}
Figure