LightFuse: Lightweight CNN based Dual-exposure Fusion

Ziyi Liu, Jie Yang, Orly Yadid-PechtSSRN 2021 · DOI · OpenReview

TL;DR

This paper introduces LightFuse, a novel lightweight Convolutional Neural Network (CNN) architecture for High Dynamic Range (HDR) image fusion from dual-exposure inputs. LightFuse significantly reduces computational cost and parameter count compared to existing methods, making it suitable for deployment on resource-constrained embedded systems without sacrificing image quality.

What Problem Does This Paper Address?

Capturing scenes with a wide dynamic range (the difference between the brightest and darkest parts) is a challenge for standard digital cameras, often resulting in overexposed or underexposed areas that lose detail. High Dynamic Range (HDR) imaging aims to overcome this by merging multiple exposures. While deep learning, particularly CNNs, has shown great promise in HDR image fusion, existing CNN-based methods are often computationally intensive and require significant memory, making them impractical for embedded systems like smartphones or drones. This paper addresses the need for an efficient yet effective HDR fusion method that can run on devices with limited power and hardware resources.

Methods

LightFuse employs a novel architecture consisting of two specialized sub-networks:

  • GlobalNet (G): This network is designed to capture the global structural information and illumination across the scene. It uses a U-Net-like encoder-decoder structure but omits skip connections to reduce memory overhead, focusing on spatial dimension processing.
  • DetailNet (D): This network is responsible for enhancing local details and fine-grained information. It utilizes a series of depthwise separable convolutions (DS_Conv) and pointwise convolutions (P_Conv) to efficiently process information across channels. The paper highlights that DS_Conv significantly reduces computation and parameters compared to standard convolutions by decoupling spatial and channel-wise operations. DetailNet is designed for efficient tile-based processing to minimize off-chip memory access.

The outputs of GlobalNet and DetailNet are element-wise added and passed through a Tanh activation function to produce the final fused HDR image. The loss function combines perceptual loss (using VGG19) and Mean Squared Error (MSE) to ensure both high-level feature similarity and pixel-wise accuracy.

Datasets

The SICE dataset [39] is used for training and testing. This dataset contains multi-exposure image sequences. For this work, the authors select pairs of images with the highest and lowest exposure values from each scene to form the dual-exposure input. They also mention cropping images into 256x256 patches for training, resulting in approximately 68,000 patches.

Key Insights and Ideas

  • Decomposition of Fusion Task: Separating the fusion task into global illumination (GlobalNet) and local details (DetailNet) allows for specialized processing and improved efficiency.
  • Depthwise Separable Convolutions (DS_Conv) for Efficiency: The paper strongly advocates for DS_Conv as a fundamental building block for lightweight CNNs, demonstrating its significant reduction in parameters and computations while maintaining competitive performance.
  • Tile-Based Processing in DetailNet: Implementing tile-based processing in DetailNet is crucial for reducing off-chip memory bandwidth requirements, a key bottleneck in embedded systems.
  • Feature Map Addition: Using element-wise addition of feature maps from GlobalNet and DetailNet is presented as a computationally efficient way to merge information, similar to residual learning principles.

Key Results

  • Performance: LightFuse achieves competitive HDR image fusion quality, outperforming state-of-the-art methods in PSNR by up to 8.7 and SSIM by a notable margin.
  • Efficiency: The model demonstrates a dramatic reduction in parameters, being 16.7x to 306.2x smaller than other methods. It also achieves significant speedups, running 3.5x to 6508x faster than other algorithms on the same testing set.
  • Embedded Deployment: The paper successfully demonstrates the deployment of LightFuse on a Raspberry Pi 2 and an FPGA (Avnet ZedBoard), achieving real-time performance (30 frames per second on FPGA) and showing visually pleasing results.

Limitations and Future Work

  • Limitations: The paper acknowledges that while LightFuse performs well, it might not be as effective as methods requiring three or more exposure inputs for certain complex scenes. The performance on the Raspberry Pi is shown, but a detailed quantitative comparison on this platform is not extensively presented.
  • Future Work: The authors plan to deploy LightFuse on other novel platforms and explore its application to other HDR imaging challenges, such as tone mapping operators and single-image enhancement, using lightweight neural networks.

Summary updated

Abstract

Deep convolutional neural networks (DCNNs) have aided high dynamic range (HDR) imaging recently and have received a lot of attention. The quality of DCNN-generated HDR images has overperformed the traditional counterparts. However, DCNNs are prone to be computationally intensive and power-hungry, and hence cannot be implemented on various embedded computing platforms with limited power and hardware resources. Embedded systems have a huge market, and utilizing DCNNs' powerful functionality into them will further reduce human intervention. To address the challenge, we propose LightFuse, a lightweight CNN-based algorithm for extreme dual-exposure image fusion, which achieves better functionality than a conventional DCNN and can be deployed in embedded systems. Two sub-networks are utilized: a GlobalNet (G) and a DetailNet (D). The goal of G is to learn the global illumination information on the spatial dimension, whereas D aims to enhance local details on the channel dimension. Both G and D are based solely on depthwise convolution (D_Conv) and pointwise convolution (P_Conv) to reduce required parameters and computations. Experimental results show that this proposed technique could generate HDR images in extremely exposed regions with sufficient details to be legible. Our model outperforms other state-of-the-art approaches in peak signal-to-noise ratio (PSNR) score by 0.9 to 8.7 while achieving 16.7 to 306.2 times parameter reduction.
BibTeX
@misc{liu2021lightfuse,
  author = {Ziyi Liu and Jie Yang and Orly Yadid-Pecht},
  title = {{LightFuse: Lightweight CNN based Dual-exposure Fusion}},
  year = {2021},
  eprint = {2107.02299},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.02299},
  howpublished = {SSRN},
}
Figure