Abstract
Deep learning-based image denoising requires vast amounts of paired noisy and clean data, which is difficult to collect for diverse camera sensors. This paper introduces a hybrid framework for synthesizing realistic low-light raw noise. By combining physics-based modeling for signal-dependent noise with a GAN-based generative model for complex, signal-independent noise, the authors create high-fidelity synthetic training data. Central to this approach is a novel Fourier Transformer Discriminator (FTD) that operates in the frequency domain to better capture noise textures. Models trained with this synthetic data achieve state-of-the-art performance across multiple benchmarks and real-world sensors.
Introduction
Deep learning has revolutionized image denoising, but it faces a significant "data hunger" problem. Training a model for extreme low-light conditions typically requires thousands of pairs of short-exposure (noisy) and long-exposure (clean) images. Collecting these is a logistical challenge, requiring perfectly static scenes and precise alignment for every new camera sensor.
While physics-based models like Poisson-Gaussian distributions can synthesize noise, they are often too simplistic to capture the messy reality of readout noise, fixed pattern noise, and thermal noise found in real-world sensors. Conversely, pure deep-learning-based noise generators often struggle to model the fine-grained statistical distributions of extremely low-light environments. This paper addresses these gaps by proposing a separated noise modeling framework.
Methodology
The proposed framework splits noise synthesis into two distinct components to leverage the strengths of both physics and deep learning:
- Signal-Dependent Noise (Physics-based): Photon shot noise is well-understood and follows a Poisson distribution. The authors use a standard physics-based model for this component, as it is computationally efficient and accurate.
- Signal-Independent Noise (Learning-based): Complex sensor-specific noise (e.g., readout noise) is learned from real samples using a Generative Adversarial Network (GAN) with a U-Net architecture.
To optimize this hybrid approach, the authors introduced two key innovations:
- Domain Alignment: To ensure the GAN learns noise characteristics rather than image content, both real and synthetic noisy images are passed through a pre-trained "Denoise-Net." Loss is then calculated in this nearly noise-free space.
- Fourier Transformer Discriminator (FTD): Unlike standard discriminators that look at spatial pixels, the FTD operates in the spectral (frequency) domain. Since noise is typically high-frequency and image content is low-frequency, this allows the model to more accurately distinguish between real and synthetic noise distributions.
Datasets
The study utilizes three main benchmarks to validate the model's performance:
- SID (See-in-the-Dark): A standard dataset for extreme low-light imaging.
- ELD (Extreme Low-light Denoising): A benchmark focused on the gap between synthetic and real noise.
- LRD (Low-light Raw Denoising): A new dataset introduced by the authors containing 1,800 images across 100 scenes. Captured with a Sony IMX586 sensor, it uses Exposure Value (EV) steps rather than fixed exposure times, making it more representative of real-world photography.
Results and Discussion
The proposed method consistently outperformed existing state-of-the-art (SOTA) methods, such as ELD and Noise-Flow, across all datasets. On the SID Sony dataset at a x100 ratio, the model achieved a PSNR of 41.95 dB, significantly higher than the 37.51 dB achieved by standard physics-based models.
Key insights from the results include:
- Hybrid Superiority: Letting physics handle shot noise while the GAN handles sensor-specific noise allows the model to generalize better across different gain settings and ISO levels.
- Spectral Awareness: Using Fourier Transformers in the discriminator allows the model to match the statistical "texture" of real noise much more closely than spatial-only methods, as evidenced by the model achieving the lowest Average Kullback-Leibler Divergence (AKLD).
- Visual Fidelity: Qualitatively, denoisers trained with this synthetic noise produce images with fewer color artifacts and better-preserved textures compared to those trained on standard synthetic data.
Conclusion
This research demonstrates that a hybrid approach to noise modeling can bridge the gap between synthetic training data and real-world sensor performance. While the model currently requires a small amount of real-world noisy data for sensor-specific calibration, it represents a significant step toward generalizable low-light denoising. Future work may extend this framework to low-light video, addressing the added complexity of temporal noise consistency.
Summary updated