Multi-Aperture-Based Probabilistic Noise Reduction of Random Telegraph Signal Noise and Photon Shot Noise in Semi-Photon-Counting Complementary-Metal-Oxide-Semiconductor Image Sensor

Ishida Haruki, Keiichiro Kagawa, Takashi Komuro, Bo Zhang, Min-Woong Seo, Taishi Takasawa, Keita Yasutomi, Shoji KawahitoSensors 2018 · DOI

TL;DR

This paper introduces a novel probabilistic noise reduction technique for semi-photon-counting-level (SPCL) CMOS image sensors, which are prone to Random Telegraph Signal (RTS) noise. By leveraging a multi-aperture camera system and maximum likelihood estimation, the method effectively removes RTS noise while also reducing photon shot noise, achieving superior image quality in extremely low-light conditions.

What Problem Does This Paper Address?

Modern CMOS image sensors (CISs) are pushing towards ultra-low noise and semi-photon-counting-level (SPCL) capabilities, enabling imaging in extremely low-light environments. However, a significant hurdle remains: Random Telegraph Signal (RTS) noise. This inherent noise, caused by charge traps in the sensor's circuitry, manifests as sudden, discrete shifts in pixel values and is notoriously difficult to suppress with conventional methods like correlated double sampling (CDS). While previous approaches like selective averaging have shown promise in removing RTS noise, they often sacrifice photon shot noise reduction or introduce other artifacts. This paper aims to address the challenge of simultaneously reducing both RTS noise and photon shot noise in SPCL CISs without prior image information, thereby improving image quality in demanding low-light scenarios.

Methods

The core of the proposed solution is a multi-aperture (MA) camera system combined with Maximum Likelihood Estimation (MLE).

  • Multi-Aperture Camera: Instead of a single, high-performance lens, the system uses multiple moderately fast and compact lenses, each paired with an SPCL CIS. This setup creates redundant images of the same scene, effectively increasing the signal level and providing multiple measurements for each "synthetic" pixel. This redundancy is key to estimating and mitigating noise.
  • Noise Modeling: The paper models the sensor noise as a combination of three components: Gaussian noise (from thermal and 1/f noise), RTS noise, and photon shot noise. Each component is represented by its probability density function (PDF). RTS noise is modeled as discrete levels, and photon shot noise is assumed to follow a Poisson distribution.
  • Maximum Likelihood Estimation (MLE): The method estimates the true signal level (represented by the average number of incident photons, $\lambda$) by finding the value of $\lambda$ that maximizes the likelihood function. This likelihood function is derived from the product of the PDFs of the observed pixel values from all apertures, taking into account the modeled noise characteristics. The process involves two steps: first, characterizing the noise parameters from dark images, and second, applying MLE to the captured multi-aperture images to estimate the true signal and remove noise.

This approach differs from previous work by not discarding pixels with RTS noise (as in selective averaging) but rather by probabilistically estimating and accounting for the RTS noise level within the MLE framework.

Datasets

The paper uses simulated data for evaluation, generated based on measured real sensor noise data from an SPCL CIS. Specifically, they captured 5000 dark images to characterize the sensor's noise parameters (Gaussian and RTS noise). The ground truth images for simulation were generated by adding photon shot noise to this measured sensor noise. No new benchmark datasets were introduced.

Key Insights and Ideas

  • Redundancy as a Noise Mitigation Tool: The MA camera's inherent redundancy is not just for increasing signal but is critically leveraged to estimate and remove complex noise like RTS.
  • Probabilistic Approach to RTS Noise: Instead of simply excluding pixels affected by RTS noise, the method probabilistically models and accounts for it within an MLE framework, leading to a more robust solution.
  • Simultaneous Reduction of Multiple Noise Sources: The proposed method effectively tackles both RTS noise and photon shot noise, a significant improvement over methods that might excel at one but falter at the other.
  • Synthetic Aperture for Low-Light Imaging: The MA system effectively emulates a much faster lens (lower F-number) than is practically achievable with a single lens, opening possibilities for ultra-low-light imaging.

Key Results

The proposed multi-aperture MLE method demonstrated superior noise reduction performance compared to simple averaging and selective averaging in simulations:

  • Peak Signal-to-Noise Ratio (PSNR): The proposed method achieved the highest PSNR of 13.14 dB, closely approaching the photon shot noise limited case (14.52 dB). This is significantly better than simple averaging (11.92 dB) and selective averaging (11.61 dB).
  • Root Mean Square Error (RMSE): The proposed method also yielded the lowest RMSE of 0.42 eRMS, compared to 0.49 eRMS for simple averaging and 0.51 eRMS for selective averaging.
  • RTS Noise Suppression: Visualizations and noise histograms confirmed that the proposed method effectively removed RTS noise, while simple averaging still showed visible RTS noise, and selective averaging, though better, still had some residual RTS noise.
  • Photon Shot Noise Reduction: The method also reduced photon shot noise, bringing it close to the theoretical limit, which was not as effectively achieved by selective averaging due to pixel exclusion.

Limitations and Future Work

  • Disparity Removal: The paper acknowledges that multi-aperture images inherently contain disparities due to lens positions. While methods for disparity removal exist, they might be inaccurate in extremely low-light conditions, potentially leading to blurry denoised images. Future work is needed to quantitatively study this issue.
  • Computational Cost: The proposed MLE method is computationally intensive. For real-time applications, acceleration through parallel hardware is suggested. The authors note that current implementation took 0.78s for selective averaging and 123.14s for MLE on their simulation setup.
  • Lens Transmittance Variation: The simulation did not account for variations in lens transmittance or vignetting, which could affect the assumption of equal incident light intensity across apertures. These variations should be measured and incorporated in future processing.
  • Noise Model Generalization: While the current noise model (Gaussian and Poisson) is suitable for SPCL CISs, other image sensors might require different PDF formulations (e.g., asymmetric Gaussian, higher-order Gaussian) to accurately capture their noise characteristics.

Summary updated

Abstract

A probabilistic method to remove the random telegraph signal (RTS) noise and to increase the signal level is proposed, and was verified by simulation based on measured real sensor noise. Although semi-photon-counting-level (SPCL) ultra-low noise complementary-metal-oxide-semiconductor (CMOS) image sensors (CISs) with high conversion gain pixels have emerged, they still suffer from huge RTS noise, which is inherent to the CISs. The proposed method utilizes a multi-aperture (MA) camera that is composed of multiple sets of an SPCL CIS and a moderately fast and compact imaging lens to emulate a very fast single lens. Due to the redundancy of the MA camera, the RTS noise is removed by the maximum likelihood estimation where noise characteristics are modeled by the probability density distribution. In the proposed method, the photon shot noise is also relatively reduced because of the averaging effect, where the pixel values of all the multiple apertures are considered. An extremely low-light condition that the maximum number of electrons per aperture was the only 2 e - was simulated. PSNRs of a test image for simple averaging, selective averaging (our previous method), and the proposed method were 11.92 dB, 11.61 dB, and 13.14 dB, respectively. The selective averaging, which can remove RTS noise, was worse than the simple averaging because it ignores the pixels with RTS noise and photon shot noise was less improved. The simulation results showed that the proposed method provided the best noise reduction performance.
BibTeX
@inproceedings{haruki2018multi,
  author = {Ishida Haruki and Keiichiro Kagawa and Takashi Komuro and Bo Zhang and Min-Woong Seo and Taishi Takasawa and Keita Yasutomi and Shoji Kawahito},
  title = {{Multi-Aperture-Based Probabilistic Noise Reduction of Random Telegraph Signal Noise and Photon Shot Noise in Semi-Photon-Counting Complementary-Metal-Oxide-Semiconductor Image Sensor}},
  booktitle = {Sensors},
  year = {2018},
  doi = {10.3390/s18040977},
}
Figure