Image Denoising via Multi-Scale Gated Fusion Network

Shengyu Li, Yaowu Chen, Rongxin Jiang, Xiang TianIEEE Access 2019 · OpenReview

Abstract

Deep convolutional neural networks have made significant progress in image denoising. However, in most cases, denoising methods using a single-stream structure with a single kernel size do not perform so well in integrating complementary contextual information; owing to the lack of this type of information, they may fail to reconstruct fine textures and patterns. To address this problem, we propose a multi-scale gated fusion network (MGFN) for image denoising, which learns direct end-to-end mappings from corrupted images to clean images. Our proposed network consists of several multi-scale mutually-gated (MM) blocks. In each MM block, we incorporate dilated convolution into a merge-and-run (MR) module to exploit multi-scale features in an effective way and further recognize useful features by filtration via a gating mechanism. Moreover, we propose a simple but effective loss function named dropout-loss to train the network. The extensive experiments on benchmark datasets show that our proposed method can well recover textures, yielding favorable performance against other state-of-the-art methods.

Introduction

Image denoising is a fundamental low-level vision task aimed at recovering a clean latent image $x$ from a noisy observation $y = x + n$. While deep learning has revolutionized this field, many popular architectures rely on a single-stream structure with a fixed kernel size. These approaches often struggle to integrate complementary contextual information from different scales, which is essential for reconstructing fine textures and repetitive patterns. If a network only looks at a small local neighborhood, it might mistake a fine texture for noise and blur it away.

Proposed Method: Multi-Scale Gated Fusion Network (MGFN)

The authors introduce the Multi-Scale Gated Fusion Network (MGFN), which is built around a sequence of Multi-scale Mutually-gated (MM) blocks. The architecture features several key innovations designed to preserve image detail while removing noise.

The overall MGFN architecture (a) and the detailed structure of the Multi-scale Mutually-gated (MM) block (b).

Multi-scale Mutually-gated (MM) Blocks

The core of the MGFN is the MM block, which improves upon standard residual learning through three main mechanisms:

  1. Modified Merge-and-Run (MR) Mapping: Standard MR modules typically average inputs from parallel branches. MGFN improves this by using a learnable convolution to control how residual information is added, allowing the network to be more selective about what information it preserves.
  2. Dilated Convolutions for Multi-Scale Learning: To capture wider context without increasing the parameter count, the network uses dilated convolutions in one of its parallel branches. This allows the model to "see" larger patterns (coarse-grained info) in one path while maintaining fine-grained details in the other.
Comparison of different Merge-and-Run (MR) module configurations: (a) standard, (b) with 5x5 convolutions, and (c) the proposed multi-scale version with dilated convolutions.
  1. Mutual Gating Mechanism: This is the "secret sauce" of the MM block. The features from the left branch are used to "gate" (filter) the features of the right branch, and vice versa. This cross-pollination helps the network recognize and emphasize the most useful features for reconstruction.

Dropout-Loss

Inspired by the dropout regularization technique, the authors propose a loss function that randomly masks out pixels during the calculation of the Mean Squared Error (MSE). By only calculating loss on a subset of pixels (using a Bernoulli distribution), the network becomes more robust and avoids overfitting to specific noise patterns.

Experimental Results

The researchers evaluated MGFN across several standard benchmarks for both grayscale and color Gaussian denoising, as well as impulse noise removal, using datasets like BSD68, Set12, and Urban100.

Quantitative and Qualitative Performance

The most significant performance gains were observed on the Urban100 dataset, which is rich in complex structural patterns. Because Urban100 contains many buildings and grids, the multi-scale approach of MGFN excels where single-stream models produce blurry results. For a noise level of $\sigma=50$, MGFN achieved a PSNR of 26.74 dB, outperforming DnCNN (26.28 dB) by a substantial 0.46 dB. Visual comparisons show that MGFN preserves sharp edges and fine repetitive textures that other methods turn into gray smudges.

Efficiency

On an NVIDIA Tesla P100 GPU, MGFN can denoise a $256 \times 256$ image in just 11ms. While slightly slower than ultra-fast models like FFDNet due to its multi-path architecture, it remains highly competitive for real-time applications.

Conclusion

The Multi-Scale Gated Fusion Network addresses the limitations of single-scale denoising models by effectively fusing multi-scale features through a mutual gating mechanism. By combining dilated convolutions with a selective gating process and a novel dropout-loss function, MGFN achieves state-of-the-art results in texture recovery. Future work could involve optimizing the gating mechanism for even greater efficiency or extending the approach to other restoration tasks like super-resolution.

Summary updated

Abstract

Deep convolutional neural networks have made significant progress in image denoising. However, in most cases, denoising methods using a single-stream structure with a single kernel size do not perform so well in integrating complementary contextual information; owing to the lack of this type of information, they may fail to reconstruct fine textures and patterns. To address this problem, we propose a multi-scale gated fusion network (MGFN) for image denoising, which learns direct end-to-end mappings from corrupted images to clean images. Our proposed network consists of several multi-scale mutually-gated (MM) blocks. In each MM block, we incorporate dilated convolution into a merge-and-run (MR) module to exploit multi-scale features in an effective way and further recognize useful features by filtration via a gating mechanism. Moreover, we propose a simple but effective loss function named dropout-loss to train the network. The extensive experiments on benchmark datasets show that our proposed method can well recover textures, yielding favorable performance against other state-of-the-art methods.
BibTeX
@inproceedings{li2019image,
  author = {Shengyu Li and Yaowu Chen and Rongxin Jiang and Xiang Tian},
  title = {{Image Denoising via Multi-Scale Gated Fusion Network}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=ip5vqRAqBI8},
}
Figure