Denoising and Motion Artifact Removal Using Deformable Kernel Prediction Neural Network for Color-Intensified CMOS

Zhenghao Han, Li Li, Weiqi Jin, Xia Wang, Gangcheng Jiao, Xuan Liu, Hailin WangSensors 2021 · DOI

TL;DR

This paper introduces a novel Deformable Kernel Prediction Neural Network (DKPNN) designed to simultaneously denoise images and remove motion artifacts from color-intensified CMOS (ICMOS) sensors operating in low-light conditions. By developing a specialized data generation method and a network architecture that predicts deformable kernels, the DKPNN significantly outperforms existing methods in preserving image details and correcting color-channel misalignments inherent in time-division color imaging.

What Problem Does This Paper Address?

Traditional night-vision devices, like image intensifiers, excel in low-light but lack digital output. Intensified CMOS (ICMOS) sensors bridge this gap by coupling image intensifiers with digital sensors, offering improved performance over standard CMOS/CCD. To achieve color imaging with ICMOS, a liquid-crystal tunable filter (LCTF) is often employed in a time-division manner. However, this time-division approach, while enabling full-resolution color without demosaicing, introduces color-channel motion artifacts when there's target movement or camera shake. Existing denoising algorithms, primarily designed for single-frame or different color imaging schemes (like CFA), fail to address these specific color-channel motion artifacts effectively. This paper tackles the challenge of joint denoising and motion artifact removal for LCTF-ICMOS systems, which is crucial for applications requiring clear, color night vision.

Methods

The core of the proposed solution is the Deformable Kernel Prediction Neural Network (DKPNN). This network is designed to handle both denoising and motion artifact correction simultaneously. It comprises two main subnets:

  • Offset-Prediction Subnet: This subnet, inspired by the UNet architecture, predicts the spatial offsets for sampling pixels. Crucially, it predicts offsets relative to a three-dimensional cubic filter kernel, allowing for deformable kernel prediction. This enables the network to adapt to varying motion patterns and misalignments.
  • Weight-Prediction Subnet: This subnet takes the resampled pixels (obtained using trilinear interpolation based on the predicted offsets), the original input image, and features extracted by the offset-prediction subnet. It then predicts the weights for these resampled pixels.

The DKPNN's approach is to learn a mapping between noisy and clean image pairs, but with a key difference: it predicts deformable kernels that can adapt to the specific scene and motion artifacts. This contrasts with traditional CNNs that learn fixed kernels. The network's output is a denoised image with motion artifacts removed, achieved through a weighted average of the resampled pixels.

A significant methodological contribution is the data generation strategy for color-channel motion artifacts. This method simulates motion artifacts by extracting R, G, and B channels from adjacent frames of high-frame-rate videos and combining them into a single frame. By down-sampling video frame rates, they can generate data with varying degrees of motion severity, crucial for training the DKPNN.

Datasets

  • Training Data: The authors generated synthetic training data using the Adobe240 dataset. This dataset, containing videos with object and device motion, was processed to simulate color-channel motion artifacts at different severities by manipulating frame rates.
  • Evaluation Data:
    • Generated Test Set: This data was also generated using the Adobe240 dataset, with simulated motion artifacts and PGN (Poisson-Gaussian noise) added.
  • Real LCTF-ICMOS Noisy Data: The authors captured real-world noisy images using their experimental LCTF-ICMOS system under four different illuminant levels (ranging from 5 x 10⁻² lx to 1 x 10⁻³ lx), including both static and motion targets.

Key Insights and Ideas

  • Deformable Kernels for Motion Artifacts: The core insight is that fixed convolutional kernels are insufficient for handling the complex, scene-dependent motion artifacts in time-division color imaging. By predicting deformable kernels, the network can dynamically adapt its sampling strategy to correct for these misalignments.
  • Joint Denoising and Artifact Removal: The DKPNN effectively integrates denoising and motion artifact removal into a single, end-to-end trainable network. This is more efficient and likely more effective than a sequential approach.
  • Importance of Data Generation: The paper highlights the critical need for realistic, synthetically generated data that accurately mimics the specific color-channel motion artifacts of LCTF-ICMOS systems. This synthetic data is key to training a robust model.
  • Limitations of Standard Denoising for ICMOS: The experiments clearly show that standard denoising algorithms, even when combined with optical flow-based registration, struggle with the unique color-channel motion artifacts of ICMOS, often introducing blur or failing to correct the artifacts.

Key Results

  • Quantitative Performance: On the generated test set, the DKPNN achieved significantly higher PSNR and SSIM scores compared to methods like CBM3D, DnCNN, fDnCNN, and FFDNet (even when these were augmented with optical flow registration). For example, at a noise level of σₚ = 2 × 10⁻³, σg = 2 × 10⁻², the DKPNN achieved a PSNR of 29.33 and SSIM of 0.887, outperforming the next best method (FFDNet + FE) by a considerable margin.
  • Qualitative Performance: The DKPNN demonstrated superior ability to preserve image details and correct color-channel motion artifacts, especially in regions with fine textures or moving objects, compared to other methods.
  • Real Data Performance: While performance on real LCTF-ICMOS data was slightly degraded compared to generated data (due to differences in noise models and target complexity), the DKPNN still outperformed other methods in denoising and artifact removal, particularly in flat areas.

Limitations and Future Work

  • Noise Model Mismatch: The paper acknowledges that the PGN noise model used for training differs from the actual noise characteristics of the LCTF-ICMOS system (e.g., bright spot noise from the image intensifier). This mismatch slightly degrades the DKPNN's performance on real data.
  • Color Saturation Loss: The trilinear interpolation used for pixel resampling in the offset-prediction subnet can lead to a decrease in color saturation, particularly in areas with severe motion artifacts.
  • Target Complexity: Motion artifacts are harder to remove from targets with fewer detailed features (e.g., the toy bird in the ICMOS experiment).
  • Future Work:
    • Investigate and incorporate the specific noise model of LCTF-ICMOS into the training process.
  • Explore more advanced data augmentation techniques, potentially using generative adversarial networks.
  • Develop methods to mitigate color saturation loss, possibly through new loss functions or color restoration modules.
  • Consider removing the image intensifier's phosphor screen and using a more effective coupling method to improve signal-to-noise ratio and reduce system size.

Summary updated

Abstract

Image intensifiers are used internationally as advanced military night-vision devices. They have better imaging performance in low-light-level conditions than CMOS/CCD. The intensified CMOS (ICMOS) was developed to satisfy the digital demand of image intensifiers. In order to make the ICMOS capable of color imaging in low-light-level conditions, a liquid-crystal tunable filter based color imaging ICMOS was developed. Due to the time-division color imaging scheme, motion artifacts may be introduced when a moving target is in the scene. To solve this problem, a deformable kernel prediction neural network (DKPNN) is proposed for joint denoising and motion artifact removal, and a data generation method which generates images with color-channel motion artifacts is also proposed to train the DKPNN. The results show that, compared with other denoising methods, the proposed DKPNN performed better both on generated noisy data and on real noisy data. Therefore, the proposed DKPNN is more suitable for color ICMOS denoising and motion artifact removal. A new exploration was made for low-light-level color imaging schemes.
BibTeX
@inproceedings{han2021denoising,
  author = {Zhenghao Han and Li Li and Weiqi Jin and Xia Wang and Gangcheng Jiao and Xuan Liu and Hailin Wang},
  title = {{Denoising and Motion Artifact Removal Using Deformable Kernel Prediction Neural Network for Color-Intensified CMOS}},
  booktitle = {Sensors},
  year = {2021},
  doi = {10.3390/s21113891},
}
Figure