Seeing Motion in the Dark

Chen Chen 0003, Qifeng Chen 0001, Minh N. Do, Vladlen Koltun

Abstract

Capturing video in extreme low-light environments (sub-1 lux) presents a fundamental challenge for traditional image signal processors, often resulting in excessive noise or motion blur. This paper presents a novel deep learning pipeline and the Dark Raw Video (DRV) dataset to address these issues. By employing a Siamese network architecture trained on static scenes with a specialized self-consistency loss, the proposed method achieves temporally stable, high-quality video reconstruction that generalizes to dynamic motion.

Introduction: The Low-Light Dilemma

Traditional videography in near-total darkness is constrained by a physical trade-off: increasing ISO introduces overwhelming sensor noise, while increasing exposure time leads to unacceptable motion blur. While recent "See-in-the-Dark" (SID) research has successfully applied deep learning to single-image enhancement, extending these techniques to video introduces the critical problem of temporal instability, or flicker.

A dynamic scene captured in extreme low light, illustrating the challenges of noise and motion blur.

A primary obstacle in training video models is the lack of ground truth for dynamic scenes. While static scenes allow for long-exposure reference frames via tripods, moving subjects cannot be captured cleanly in low light. Furthermore, synthetic noise models often fail to replicate real-world sensor artifacts like quantization and non-monotonic bias in extreme darkness.

Methodology

The authors propose an end-to-end pipeline that processes raw Bayer sensor data into sRGB video. The core architecture is a ResUnet—a U-Net variant enhanced with 16 residual blocks.

To solve the temporal stability problem without dynamic ground truth, the researchers introduced a Siamese training strategy. The network is trained on pairs of different noisy frames from the same static scene. The loss function consists of two parts:

  1. Recovery Loss ($L_r$): Measures the pixel-wise difference between the network output and the long-exposure ground truth.
  2. Self-Consistency Loss ($L_c$): Minimizes the difference between the outputs of the two Siamese branches. Since the scene is static, any variation between the two processed frames is identified as unwanted flicker or noise, teaching the network to produce temporally consistent results.

The Dark Raw Video (DRV) Dataset

To support this research, the authors curated the Dark Raw Video (DRV) dataset, the first public dataset specifically for raw low-light video. Captured with a Sony RX100 VI in environments ranging from 0.5 to 5 lux, the dataset includes:

  • Static Set: 202 videos with corresponding long-exposure ground truth (120x to 300x exposure ratios).
  • Dynamic Set: Real-world videos featuring subject motion and handheld camera shake, used for perceptual evaluation.
Samples from the Dark Raw Video (DRV) dataset, showcasing the variety of indoor and outdoor scenes used for training and evaluation.

Results and Generalization

A key finding is that models trained exclusively on static data with self-consistency loss generalize remarkably well to dynamic scenes. Quantitatively, the method achieved a PSNR of 28.26 dB on static tests, outperforming per-frame SID baselines. More importantly, it achieved a Temporal PSNR (TPSNR) of 38.31, a significant jump over the 33.72 achieved without Siamese training. In human perceptual studies, the proposed method was preferred in over 84% of comparisons against state-of-the-art alternatives.

Conclusion and Future Directions

By moving the entire image signal processing (ISP) pipeline into a learned framework, the authors avoid the cascading errors of traditional pipelines, such as noise amplification during white balancing. While the system currently faces limitations in "extreme-extreme" conditions (e.g., pure moonlight at 0.01 lux) and high dynamic range preservation, it represents a major step forward in making sub-lux videography practical. Future work may integrate super-resolution to compensate for the resolution loss inherent in current noise-reduction binning techniques.

Summary updated

Abstract

Deep learning has recently been applied with impressive results to extreme low-light imaging. Despite the success of single-image processing, extreme low-light video processing is still intractable due to the difficulty of collecting raw video data with corresponding ground truth. Collecting long-exposure ground truth, as was done for single-image processing, is not feasible for dynamic scenes. In this paper, we present deep processing of very dark raw videos: on the order of one lux of illuminance. To support this line of work, we collect a new dataset of raw low-light videos, in which high-resolution raw data is captured at video rate. At this level of darkness, the signal-to-noise ratio is extremely low (negative if measured in dB) and the traditional image processing pipeline generally breaks down. A new method is presented to address this challenging problem. By carefully designing a learning-based pipeline and introducing a new loss function to encourage temporal stability, we train a siamese network on static raw videos, for which ground truth is available, such that the network generalizes to videos of dynamic scenes at test time. Experimental results demonstrate that the presented approach outperforms state-of-the-art models for burst processing, per-frame processing, and blind temporal consistency.
BibTeX
@inproceedings{chen2019seeing,
  author = {Chen Chen and Qifeng Chen and Minh N. Do and Vladlen Koltun},
  title = {{Seeing Motion in the Dark}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00328},
}
Figure