Abstract
Capturing video in extreme low-light environments (sub-1 lux) presents a fundamental challenge for traditional image signal processors, often resulting in excessive noise or motion blur. This paper presents a novel deep learning pipeline and the Dark Raw Video (DRV) dataset to address these issues. By employing a Siamese network architecture trained on static scenes with a specialized self-consistency loss, the proposed method achieves temporally stable, high-quality video reconstruction that generalizes to dynamic motion.
Introduction: The Low-Light Dilemma
Traditional videography in near-total darkness is constrained by a physical trade-off: increasing ISO introduces overwhelming sensor noise, while increasing exposure time leads to unacceptable motion blur. While recent "See-in-the-Dark" (SID) research has successfully applied deep learning to single-image enhancement, extending these techniques to video introduces the critical problem of temporal instability, or flicker.
A primary obstacle in training video models is the lack of ground truth for dynamic scenes. While static scenes allow for long-exposure reference frames via tripods, moving subjects cannot be captured cleanly in low light. Furthermore, synthetic noise models often fail to replicate real-world sensor artifacts like quantization and non-monotonic bias in extreme darkness.
Methodology
The authors propose an end-to-end pipeline that processes raw Bayer sensor data into sRGB video. The core architecture is a ResUnet—a U-Net variant enhanced with 16 residual blocks.
To solve the temporal stability problem without dynamic ground truth, the researchers introduced a Siamese training strategy. The network is trained on pairs of different noisy frames from the same static scene. The loss function consists of two parts:
- Recovery Loss ($L_r$): Measures the pixel-wise difference between the network output and the long-exposure ground truth.
- Self-Consistency Loss ($L_c$): Minimizes the difference between the outputs of the two Siamese branches. Since the scene is static, any variation between the two processed frames is identified as unwanted flicker or noise, teaching the network to produce temporally consistent results.
The Dark Raw Video (DRV) Dataset
To support this research, the authors curated the Dark Raw Video (DRV) dataset, the first public dataset specifically for raw low-light video. Captured with a Sony RX100 VI in environments ranging from 0.5 to 5 lux, the dataset includes:
- Static Set: 202 videos with corresponding long-exposure ground truth (120x to 300x exposure ratios).
- Dynamic Set: Real-world videos featuring subject motion and handheld camera shake, used for perceptual evaluation.
Results and Generalization
A key finding is that models trained exclusively on static data with self-consistency loss generalize remarkably well to dynamic scenes. Quantitatively, the method achieved a PSNR of 28.26 dB on static tests, outperforming per-frame SID baselines. More importantly, it achieved a Temporal PSNR (TPSNR) of 38.31, a significant jump over the 33.72 achieved without Siamese training. In human perceptual studies, the proposed method was preferred in over 84% of comparisons against state-of-the-art alternatives.
Conclusion and Future Directions
By moving the entire image signal processing (ISP) pipeline into a learned framework, the authors avoid the cascading errors of traditional pipelines, such as noise amplification during white balancing. While the system currently faces limitations in "extreme-extreme" conditions (e.g., pure moonlight at 0.01 lux) and high dynamic range preservation, it represents a major step forward in making sub-lux videography practical. Future work may integrate super-resolution to compensate for the resolution loss inherent in current noise-reduction binning techniques.
Summary updated