Enhancing Low Light Videos by Exploring High Sensitivity Camera Noise

Wei Wang, Xin Chen, Cheng Yang, Xiang Li, Xuemei Hu, Tao YueICCV 2019 · OpenReview

Abstract

Enhancing low light videos, which consists of denoising and brightness adjustment, is an intriguing but knotty problem. Under low light condition, due to high sensitivity camera setting, commonly negligible noises become obvious and severely deteriorate the captured videos. To recover high quality videos, a mass of image/video denoising/enhancing algorithms are proposed, most of which follow a set of simple assumptions about the statistic characters of camera noise, e.g., independent and identically distributed(i.i.d.), white, additive, Gaussian, Poisson or mixture noises. However, the practical noise under high sensitivity setting in real captured videos is complex and inaccurate to model with these assumptions. In this paper, we explore the physical origins of the practical high sensitivity noise in digital cameras, model them mathematically, and propose to enhance the low light videos based on the noise model by using an LSTM-based neural network. Specifically, we generate the training data with the proposed noise model and train the network with the dark noisy video as input and clear-bright video as output. Extensive comparisons on both synthetic and real captured low light videos with the state-of-the-art methods are conducted to demonstrate the effectiveness of the proposed method.

Introduction

Capturing video in extremely low light requires high ISO settings, which amplify subtle sensor imperfections into glaring visual artifacts. Most existing denoising algorithms assume noise is "i.i.d." (independent and identically distributed) white Gaussian or Poisson noise. However, real-world high-sensitivity noise is far more complex, characterized by dynamic streaks, color shifts, and non-linear effects that standard models fail to capture. This leads to enhanced videos that remain muddy or visually inconsistent.

Physically-Grounded Noise Modeling

To address these shortcomings, this research moves beyond simple Gaussian noise models to propose a physically-grounded noise model for high-ISO low-light imaging. The model accounts for three primary components:

  1. Dynamic Streak Noise (DSN): Horizontal or vertical lines that fluctuate frame-by-frame. The authors identify that for rolling shutter cameras, this follows a 1/f (flicker) noise distribution, whereas global shutter cameras exhibit white noise characteristics.
  2. Channel Heterogeneity: Noise levels vary significantly across R, G, and B channels, often leading to a purplish color cast in boosted images.
  3. Clipping Effects: Loss of information occurs when signal levels fall below the sensor's noise floor (zero-clipping), introducing statistical bias.
Comparison of real and synthetic noise components, including dynamic streaks, channel heterogeneity, and clipping effects.

The model is made practical through camera-specific parameter calibration using dark-field images. By estimating system gain and read noise variance, the model can synthesize highly realistic training data tailored to specific hardware, such as the Canon 5D Mark III.

Enhancement via ST-LSTM

The proposed enhancement framework utilizes a Spatio-Temporal LSTM (ST-LSTM) architecture. Unlike standard CNNs, the ST-LSTM maintains a memory state that flows both vertically (across layers) and horizontally (across time frames). This allows the network to adaptively extract features and maintain temporal consistency, which is vital for preventing "flicker" in enhanced video. The network is trained using pairs of "dark/noisy" videos synthesized with the proposed model and their corresponding "clean/bright" ground truths.

Experimental Results

Extensive evaluations on both synthetic and real-world datasets demonstrate the effectiveness of the approach:

  • Quantitative Performance: On synthetic test sets, the proposed method outperformed state-of-the-art methods like TOFlow by at least 2dB in PSNR and 0.05 in SSIM.
  • Qualitative Improvement: In real-world low-light tests (ISO 25600), the method successfully removed heavy horizontal streaks and corrected the "purplish" color cast common in boosted low-light images.
  • Ablation Study: Results confirmed that the improvement stems largely from the realistic noise model; training the same ST-LSTM on standard AWGN or mixture models yielded significantly inferior results.

Conclusion

By exploring the physical origins of high-sensitivity camera noise, this work provides a robust foundation for low-light video enhancement. The integration of a realistic noise model with a spatio-temporal learning architecture effectively addresses the "knotty" artifacts of real-world sensors. Future work will focus on blind estimation of noise parameters directly from noisy inputs, eliminating the need for pre-calibration steps.

Summary updated

Abstract

Enhancing low light videos, which consists of denoising and brightness adjustment, is an intriguing but knotty problem. Under low light condition, due to high sensitivity camera setting, commonly negligible noises become obvious and severely deteriorate the captured videos. To recover high quality videos, a mass of image/video denoising/enhancing algorithms are proposed, most of which follow a set of simple assumptions about the statistic characters of camera noise, e.g., independent and identically distributed(i.i.d.), white, additive, Gaussian, Poisson or mixture noises. However, the practical noise under high sensitivity setting in real captured videos is complex and inaccurate to model with these assumptions. In this paper, we explore the physical origins of the practical high sensitivity noise in digital cameras, model them mathematically, and propose to enhance the low light videos based on the noise model by using an LSTM-based neural network. Specifically, we generate the training data with the proposed noise model and train the network with the dark noisy video as input and clear-bright video as output. Extensive comparisons on both synthetic and real captured low light videos with the state-of-the-art methods are conducted to demonstrate the effectiveness of the proposed method.
BibTeX
@inproceedings{wang2019enhancing,
  author = {Wei Wang and Xin Chen and Cheng Yang and Xiang Li and Xuemei Hu and Tao Yue},
  title = {{Enhancing Low Light Videos by Exploring High Sensitivity Camera Noise}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=WFxiaD2bt3},
}
Figure