Iterative Residual Network for Deep Joint Image Demosaicking and Denoising

Filippos Kokkinos, Stamatios Lefkimmiatis

Abstract

Modern digital cameras rely on the sequential execution of separate image processing steps to produce realistic images. The first two steps are usually related to denoising and demosaicking where the former aims to reduce noise from the sensor and the latter converts a series of light intensity readings to color images. Modern approaches try to jointly solve these problems, i.e. joint denoising-demosaicking which is an inherently ill-posed problem given that two-thirds of the intensity information is missing and the rest are perturbed by noise. While there are several machine learning systems that have been recently introduced to solve this problem, the majority of them relies on generic network architectures which do not explicitly take into account the physical image model. In this work we propose a novel algorithm which is inspired by powerful classical image regularization methods, large-scale optimization, and deep learning techniques. Consequently, our derived iterative optimization algorithm, which involves a trainable denoising network, has a transparent and clear interpretation compared to other black-box data driven approaches. Our extensive experimentation line demonstrates that our proposed method outperforms any previous approaches for both noisy and noise-free data across many different datasets. This improvement in reconstruction quality is attributed to the rigorous derivation of an iterative solution and the principled way we design our denoising network architecture, which as a result requires fewer trainable parameters than the current state-of-the-art solution and furthermore can be efficiently trained by using a significantly smaller number of training data than existing deep demosaicking networks.

Introduction

Digital camera sensors capture raw data through a Color Filter Array (CFA), where each pixel records only one color component (Red, Green, or Blue). To reconstruct a full-color image, two critical steps are required: demosaicking (interpolating the missing color values) and denoising (removing sensor noise). Traditionally, these tasks are performed sequentially. However, this approach is suboptimal: denoising before demosaicking can blur details necessary for accurate interpolation, while demosaicking before denoising can transform simple sensor noise into complex artifacts.

While recent deep learning models have attempted joint denoising-demosaicking, many are "black-box" architectures that require massive datasets and are often hard-coded for specific CFA patterns, such as the Bayer pattern. This paper introduces a novel iterative framework that bridges classical optimization with deep learning to address these limitations.

The Proposed MMNet Framework

The authors propose MMNet, an architecture derived from the Majorization-Minimization (MM) optimization framework. By formulating the problem as a Maximum A Posteriori (MAP) estimation, the algorithm decouples the physical image model (the CFA mask) from the image prior (the denoiser).

The iterative framework of the proposed MMNet, showing the unrolled denoising steps.

Key features of the architecture include:

  • Unrolled Iterative Structure: The optimization process is "unrolled" into $K$ iterations. Each step updates the image based on the physical data and passes it through a shared Residual Denoising Network (ResDNet).
  • Weight Sharing: Unlike typical unrolled networks, MMNet shares parameters across all iterations, reducing the parameter count to approximately 380k—significantly lower than the >1.2M parameters in competing models.
  • Noise-Awareness: The ResDNet module explicitly accepts the noise variance ($\sigma$) as an input, allowing a single model to generalize across various noise levels.
  • Efficient Training: To manage memory during training with many iterations, the authors utilize Truncated Back-propagation Through Time (TBPTT).

Experimental Results

The model was trained on the Microsoft Demosaicking and MIT datasets and evaluated on standard benchmarks including Kodak, McMaster (McM), VDP, and Moire.

The results demonstrate that MMNet achieves state-of-the-art performance. On the noisy MSR dataset, it reached a PSNR of 40.1 dB, surpassing the previous best of 38.8 dB. Qualitatively, the method effectively reduces common artifacts like "zippering" and moiré patterns in high-frequency regions.

Quantitative comparison of MMNet against state-of-the-art methods across standard datasets.

Conclusion and Key Insights

MMNet offers a transparent and efficient alternative to black-box deep learning models. By treating the CFA pattern as a variable in the optimization step, the architecture is inherently CFA-agnostic, making it applicable to any sensor pattern (e.g., Fuji X-Trans) without modification. Furthermore, the iterative nature allows for a flexible trade-off between speed and quality; with just two iterations, the model is twice as fast as competing deep networks while maintaining high reconstruction quality.

Summary updated

Abstract

Modern digital cameras rely on the sequential execution of separate image processing steps to produce realistic images. The first two steps are usually related to denoising and demosaicking where the former aims to reduce noise from the sensor and the latter converts a series of light intensity readings to color images. Modern approaches try to jointly solve these problems, i.e. joint denoising-demosaicking which is an inherently ill-posed problem given that two-thirds of the intensity information is missing and the rest are perturbed by noise. While there are several machine learning systems that have been recently introduced to solve this problem, the majority of them relies on generic network architectures which do not explicitly take into account the physical image model. In this work we propose a novel algorithm which is inspired by powerful classical image regularization methods, large-scale optimization, and deep learning techniques. Consequently, our derived iterative optimization algorithm, which involves a trainable denoising network, has a transparent and clear interpretation compared to other black-box data driven approaches. Our extensive experimentation line demonstrates that our proposed method outperforms any previous approaches for both noisy and noise-free data across many different datasets. This improvement in reconstruction quality is attributed to the rigorous derivation of an iterative solution and the principled way we design our denoising network architecture, which as a result requires fewer trainable parameters than the current state-of-the-art solution and furthermore can be efficiently trained by using a significantly smaller number of training data than existing deep demosaicking networks. Code and results can be found at https://github.com/cig-skoltech/deep_demosaick
BibTeX
@article{kokkinos2019iterative,
  author = {Filippos Kokkinos and Stamatios Lefkimmiatis},
  title = {{Iterative Residual Network for Deep Joint Image Demosaicking and Denoising}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=s8nNfnevEA},
}
Figure