Towards General Low-Light Raw Noise Synthesis and Modeling

Feng Zhang, Bin Xu, Zhiqiang Li, Xinran Liu, Qingbo Lu, Changxin Gao, Nong Sang

Abstract

Deep learning-based image denoising requires vast amounts of paired noisy and clean data, which is difficult to collect for diverse camera sensors. This paper introduces a hybrid framework for synthesizing realistic low-light raw noise. By combining physics-based modeling for signal-dependent noise with a GAN-based generative model for complex, signal-independent noise, the authors create high-fidelity synthetic training data. Central to this approach is a novel Fourier Transformer Discriminator (FTD) that operates in the frequency domain to better capture noise textures. Models trained with this synthetic data achieve state-of-the-art performance across multiple benchmarks and real-world sensors.

Introduction

Deep learning has revolutionized image denoising, but it faces a significant "data hunger" problem. Training a model for extreme low-light conditions typically requires thousands of pairs of short-exposure (noisy) and long-exposure (clean) images. Collecting these is a logistical challenge, requiring perfectly static scenes and precise alignment for every new camera sensor.

While physics-based models like Poisson-Gaussian distributions can synthesize noise, they are often too simplistic to capture the messy reality of readout noise, fixed pattern noise, and thermal noise found in real-world sensors. Conversely, pure deep-learning-based noise generators often struggle to model the fine-grained statistical distributions of extremely low-light environments. This paper addresses these gaps by proposing a separated noise modeling framework.

Methodology

The proposed framework splits noise synthesis into two distinct components to leverage the strengths of both physics and deep learning:

  1. Signal-Dependent Noise (Physics-based): Photon shot noise is well-understood and follows a Poisson distribution. The authors use a standard physics-based model for this component, as it is computationally efficient and accurate.
  2. Signal-Independent Noise (Learning-based): Complex sensor-specific noise (e.g., readout noise) is learned from real samples using a Generative Adversarial Network (GAN) with a U-Net architecture.
The proposed Separated Noise Modeling framework, combining physics-based Poisson sampling with a GAN-based generator and a Fourier Transformer Discriminator.

To optimize this hybrid approach, the authors introduced two key innovations:

  • Domain Alignment: To ensure the GAN learns noise characteristics rather than image content, both real and synthetic noisy images are passed through a pre-trained "Denoise-Net." Loss is then calculated in this nearly noise-free space.
  • Fourier Transformer Discriminator (FTD): Unlike standard discriminators that look at spatial pixels, the FTD operates in the spectral (frequency) domain. Since noise is typically high-frequency and image content is low-frequency, this allows the model to more accurately distinguish between real and synthetic noise distributions.

Datasets

The study utilizes three main benchmarks to validate the model's performance:

  • SID (See-in-the-Dark): A standard dataset for extreme low-light imaging.
  • ELD (Extreme Low-light Denoising): A benchmark focused on the gap between synthetic and real noise.
  • LRD (Low-light Raw Denoising): A new dataset introduced by the authors containing 1,800 images across 100 scenes. Captured with a Sony IMX586 sensor, it uses Exposure Value (EV) steps rather than fixed exposure times, making it more representative of real-world photography.

Results and Discussion

The proposed method consistently outperformed existing state-of-the-art (SOTA) methods, such as ELD and Noise-Flow, across all datasets. On the SID Sony dataset at a x100 ratio, the model achieved a PSNR of 41.95 dB, significantly higher than the 37.51 dB achieved by standard physics-based models.

Visual and quantitative comparison of the proposed method against state-of-the-art denoising techniques on the SID dataset.

Key insights from the results include:

  • Hybrid Superiority: Letting physics handle shot noise while the GAN handles sensor-specific noise allows the model to generalize better across different gain settings and ISO levels.
  • Spectral Awareness: Using Fourier Transformers in the discriminator allows the model to match the statistical "texture" of real noise much more closely than spatial-only methods, as evidenced by the model achieving the lowest Average Kullback-Leibler Divergence (AKLD).
  • Visual Fidelity: Qualitatively, denoisers trained with this synthetic noise produce images with fewer color artifacts and better-preserved textures compared to those trained on standard synthetic data.

Conclusion

This research demonstrates that a hybrid approach to noise modeling can bridge the gap between synthetic training data and real-world sensor performance. While the model currently requires a small amount of real-world noisy data for sensor-specific calibration, it represents a significant step toward generalizable low-light denoising. Future work may extend this framework to low-light video, addressing the added complexity of temporal noise consistency.

Summary updated

Abstract

Modeling and synthesizing low-light raw noise is a fundamental problem for computational photography and image processing applications. Although most recent works have adopted physics-based models to synthesize noise, the signal-independent noise in low-light conditions is far more complicated and varies dramatically across camera sensors, which is beyond the description of these models. To address this issue, we introduce a new perspective to synthesize the signal-independent noise by a generative model. Specifically, we synthesize the signal-dependent and signal-independent noise in a physics- and learning-based manner, respectively. In this way, our method can be considered as a general model, that is, it can simultaneously learn different noise characteristics for different ISO levels and generalize to various sensors. Subsequently, we present an effective multi-scale discriminator termed Fourier transformer discriminator (FTD) to distinguish the noise distribution accurately. Additionally, we collect a new low-light raw denoising (LRD) dataset for training and benchmarking. Qualitative validation shows that the noise generated by our proposed noise model can be highly similar to the real noise in terms of distribution. Furthermore, extensive denoising experiments demonstrate that our method performs favorably against state-of-the-art methods on different sensors.
BibTeX
@inproceedings{zhang2023towards,
  author = {Feng Zhang and Bin Xu and Zhiqiang Li and Xinran Liu and Qingbo Lu and Changxin Gao and Nong Sang},
  title = {{Towards General Low-Light Raw Noise Synthesis and Modeling}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00993},
}
Figure