Megapixel Photon-Counting Color Imaging using Quanta Image Sensor

Abhiram Gnanasambandam, Omar A. Elgendy, Jiaju Ma, Stanley H. ChanarXiv 2019 · DOI · OpenReview

TL;DR

This paper presents the first megapixel color Quanta Image Sensor (QIS) capable of single-photon counting, enabling color imaging in extremely low-light conditions. They introduce a novel joint reconstruction and demosaicing algorithm that leverages variance stabilizing transforms and Plug-and-Play ADMM to overcome the challenges of photon shot noise and sparse data.

What Problem Does This Paper Address?

Traditional CMOS image sensors struggle in low-light conditions due to photon shot noise and limited pixel capacity. While single-photon detectors like SPADs offer high sensitivity, achieving high-resolution color imaging with them has been difficult due to pixel size constraints and cross-talk issues. This paper tackles the challenge of enabling high-resolution, low-light color imaging by demonstrating a megapixel QIS and developing a specialized reconstruction algorithm to process its unique data.

Methods

The core of the paper lies in two main innovations:

  • Megapixel Color QIS: The authors present a 1024x1024 pixel QIS with an on-chip color filter array. This sensor supports both single-bit (binary photon detection) and multi-bit (counting up to a threshold) photon counting, offering a significant leap in resolution for single-photon detectors.
  • Joint Reconstruction and Demosaicing Algorithm: To process the QIS data, they developed a custom algorithm that combines reconstruction and demosaicing. Key components include:
  • Variance Stabilizing Transform (VST): This statistical technique transforms the photon count data (which follows Poisson or Binomial distributions) into a form that approximates Gaussian noise, making it compatible with standard denoising algorithms.
  • Plug-and-Play ADMM (PnP ADMM): This iterative optimization framework is used for joint reconstruction and denoising. It leverages an off-the-shelf image denoiser (like BM3D) within the ADMM framework, allowing for efficient recovery of the full-color image from the sparse, noisy QIS measurements.
  • Non-iterative Algorithm: For applications where speed is paramount, they also propose a faster, non-iterative method that trades off some spatial resolution (using four jots per pixel) to bypass the iterative steps.

Datasets

The paper evaluates their method using:

  • Simulated QIS data: They simulate image formation pipelines to generate synthetic QIS raw input data with varying numbers of bits and photon counts. This allows for quantitative evaluation and comparison with ground truth.
  • Real QIS data: They present experimental results using their prototype QIS camera module on a Digital SG ColorChecker chart and real-world scenes (e.g., "QIS" sign, mushroom, duck).

Key Insights and Ideas

  • Color Imaging at the Photon Frontier: The paper demonstrates that high-quality color imaging is achievable even at extremely low photon counts (a few photons per pixel), a regime where conventional sensors fail.
  • VST is Crucial for Low-Light Color Reconstruction: The variance stabilizing transform is essential for making the photon-counting data amenable to standard image processing techniques like denoising, which are typically designed for Gaussian noise.
  • Joint Reconstruction and Demosaicing is Necessary: Simply applying standard demosaicing to QIS data would be insufficient due to the inherent noise and sparsity. A joint approach that considers both color reconstruction and denoising simultaneously is required.
  • Trade-off between Resolution and Speed: The non-iterative algorithm highlights a practical trade-off: sacrificing some spatial resolution can lead to significant speed improvements, making the technology more viable for real-time applications.

Key Results

  • Superior Low-Light Performance: The QIS, combined with their algorithm, demonstrates superior color imaging performance in very low-light conditions compared to simulated conventional CMOS sensors.
  • Quantitative Evaluation: Synthetic experiments show that their method achieves good visual quality and quantitative metrics (e.g., PSNR) that are competitive with or better than existing methods, especially at low photon counts.
  • Real-World Demonstrations: The paper showcases impressive reconstructed color images from their megapixel QIS, including a Digital SG ColorChecker chart and various scenes, highlighting the practical applicability of their system.
  • Speed Improvement: The non-iterative algorithm offers a substantial speedup (e.g., 4 minutes for iterative vs. 10 seconds for non-iterative) while maintaining comparable visual quality.

Limitations and Future Work

  • Multi-bit Saturation: The authors acknowledge that their current model assumes multi-bit saturation does not occur. They suggest implementing automatic exposure control to manage this in future work.
  • Color Cross-talk: While they use a color correction matrix to mitigate cross-talk, it's an area that could be further refined.
  • Hardware Integration: The paper mentions that the temporal binning step could potentially be integrated into the camera hardware for further efficiency.
  • Advanced Denoisers: The non-iterative algorithm could benefit from using more advanced denoisers than BM3D.
  • Real-time Applications: Further optimization of the algorithms and hardware integration are needed to achieve true real-time color imaging at these low light levels.

Summary updated

Abstract

Quanta Image Sensor (QIS) is a single-photon detector designed for extremely low light imaging conditions. Majority of the existing QIS prototypes are monochrome based on single-photon avalanche diodes (SPAD). Passive color imaging has not been demonstrated with single-photon detectors due to the intrinsic difficulty of shrinking the pixel size and increasing the spatial resolution while maintaining acceptable intra-pixel cross-talk. In this paper, we present image reconstruction of the first color QIS with a resolution of $1024 \times 1024$ pixels, supporting both single-bit and multi-bit photon counting capability. Our color image reconstruction is enabled by a customized joint demosaicing-denoising algorithm, leveraging truncated Poisson statistics and variance stabilizing transforms. Experimental results of the new sensor and algorithm demonstrate superior color imaging performance for very low-light conditions with a mean exposure of as low as a few photons per pixel in both real and simulated images.
BibTeX
@misc{gnanasambandam2019megapixel,
  author = {Abhiram Gnanasambandam and Omar A. Elgendy and Jiaju Ma and Stanley H. Chan},
  title = {{Megapixel Photon-Counting Color Imaging using Quanta Image Sensor}},
  year = {2019},
  eprint = {1903.09036},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.09036},
}
Figure