Color-Dense Illumination Adjustment Network for Removing Haze and Smoke from Fire Scenario Images

Chuansheng Wang, Jinxing Hu, Xiaowei Luo, Mei‐Po Kwan, Weihua Chen, Hao WangSensors 2022 · DOI

TL;DR

This paper introduces CIANet, a novel deep learning network designed to remove haze and smoke from images captured in fire scenarios. By proposing a new "Aerosol Scattering Model" (ESM) and a multi-branch network architecture, CIANet effectively recovers image clarity and color, outperforming existing methods and improving downstream vision tasks like object detection.

What Problem Does This Paper Address?

Images captured in fire scenarios are often severely degraded by smoke and haze, significantly reducing visibility. This poses a major challenge for robotic rescue operations and monitoring systems that rely on clear visual input. Traditional image dehazing methods, often based on the Atmospheric Scattering Model (ASM), struggle with the complex and variable nature of fire-related aerosols, leading to inaccurate estimations of atmospheric light and transmission. These limitations hinder the performance of high-level computer vision algorithms in such critical environments.

Methods

The core of this work is the Color-Dense Illumination Adjustment Network (CIANet). Unlike previous approaches that might directly map hazy images to clear ones, CIANet employs a three-branch architecture. This network jointly estimates:

  • Transmission map (t(x)): How much light reaches the camera.
  • Atmospheric light (a): The overall ambient light.
  • Aerosol color (c(x)): The dominant color of the smoke/haze.
  • Preliminary recovered scene (JDirect): An initial estimate of the clear image.

A key innovation is the Aerosol Scattering Model (ESM), a modification of the traditional ASM. ESM explicitly incorporates the color of aerosols (c(x)) into the imaging model, moving beyond the ASM's assumption of white haze. This allows for more accurate modeling of the complex visual degradation in fire scenarios. The ESM is then used to refine the output from CIANet's J-decoder. The network architecture itself utilizes a shared encoder and three separate decoders, leveraging techniques like Res2Net for multi-scale feature extraction.

Datasets

  • RFSIE (Realistic Fire Single Image Enhancement): A newly created dataset for training and evaluation, consisting of 500 fire scenario images with low degradation, augmented to create 5000 hazy images.
  • NTIRE'20 competition dataset: Used for training and evaluation, known for its non-homogeneous haze and relevance to fire scenarios.

Key Insights and Ideas

  • The color of haze matters: The assumption of white haze in traditional ASM is a significant limitation for fire scenarios, where smoke color can vary widely. ESM's explicit modeling of aerosol color is crucial for accurate dehazing.
  • Joint estimation is powerful: Instead of estimating transmission and atmospheric light separately, CIANet's multi-branch approach jointly recovers transmission, atmospheric light, and aerosol color, leading to more robust results.
  • ESM improves realism: The proposed ESM, by incorporating aerosol color, produces more visually pleasing and accurate dehazed images compared to traditional ASM, especially in challenging scenarios with dark aerosols.
  • CIANet benefits downstream tasks: The improved image quality from CIANet significantly boosts the performance of other computer vision applications, such as object detection and local keypoint matching.

Key Results

  • Quantitative Performance: CIANet achieved superior PSNR and SSIM scores on both RFSIE and NTIRE'20 datasets compared to state-of-the-art methods. For instance, the JESM output (CIANet with ESM) achieved a PSNR of 18.34 dB and SSIM of 0.62 on RFSIE, significantly outperforming JDirect (13.11 dB, 0.56) and JASM (14.23 dB, 0.58).
  • Qualitative Performance: Visual comparisons showed CIANet producing clearer images with better color restoration and fewer artifacts, especially in challenging real-world fire scenarios with uneven haze and highlighted areas.
  • Object Detection Improvement: When used as a pre-processing step for Faster R-CNN, CIANet led to a significant improvement in detection performance.
  • Runtime Efficiency: CIANet exhibits a promising runtime efficiency, with an average processing time of 4.77 seconds per image on a CPU, making it suitable for real-time applications.

Limitations and Future Work

  • Limitations: While CIANet shows robustness, the paper notes that the design of the loss function, though simple, is not the primary focus and could potentially be further optimized. The paper also acknowledges that the traditional ASM has limitations when the scene is inherently similar to airlight.
  • Future Work: The authors suggest extending their research to image enhancement for lidar images in fire scenarios, aiming to address situations where visual information is heavily obscured by smoke. They also propose further investigation into improving object detection and local keypoint matching tasks using their dehazing approach.

Summary updated

Abstract

The atmospheric particles and aerosols from burning usually cause visual artifacts in single images captured from fire scenarios. Most existing haze removal methods exploit the atmospheric scattering model (ASM) for visual enhancement, which inevitably leads to inaccurate estimation of the atmosphere light and transmission matrix of the smoky and hazy inputs. To solve these problems, we present a novel color-dense illumination adjustment network (CIANet) for joint recovery of transmission matrix, illumination intensity, and the dominant color of aerosols from a single image. Meanwhile, to improve the visual effects of the recovered images, the proposed CIANet jointly optimizes the transmission map, atmospheric optical value, the color of aerosol, and a preliminary recovered scene. Furthermore, we designed a reformulated ASM, called the aerosol scattering model (ESM), to smooth out the enhancement results while keeping the visual effects and the semantic information of different objects. Experimental results on both the proposed RFSIE and NTIRE'20 demonstrate our superior performance favorably against state-of-the-art dehazing methods regarding PSNR, SSIM and subjective visual quality. Furthermore, when concatenating CIANet with Faster R-CNN, we witness an improvement of the objection performance with a large margin.
BibTeX
@inproceedings{wang2022color,
  author = {Chuansheng Wang and Jinxing Hu and Xiaowei Luo and Mei‐Po Kwan and Weihua Chen and Hao Wang},
  title = {{Color-Dense Illumination Adjustment Network for Removing Haze and Smoke from Fire Scenario Images}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22030911},
}
Figure