Learning Temporal Consistency for Low Light Video Enhancement From Single Images

Fan Zhang, Yu Li, Shaodi You, Ying FuCVPR 2021 · OpenReview

Abstract

Single image low light enhancement is an important task and it has many practical applications. Most existing methods adopt a single image approach. Although their performance is satisfying on a static single image, we found, however, they suffer serious temporal instability when handling low light videos. We notice the problem is because existing data-driven methods are trained from single image pairs where no temporal information is available. Unfortunately, training from real temporally consistent data is also problematic because it is impossible to collect pixel-wisely paired low and normal light videos under controlled environments in large scale and diversities with noise of identical statistics. In this paper, we propose a novel method to enforce the temporal stability in low light video enhancement with only static images. The key idea is to learn and infer motion field (optical flow) from a single image and synthesize short range video sequences. Our strategy is general and can extend to large scale datasets directly. Based on this idea, we propose our method which can infer motion prior for single image low light video enhancement and enforce temporal consistency. Rigorous experiments and user study demonstrate the state-of-the-art performance of our proposed method. Our code and model will be publicly available at https://github.com/zkawfanx/StableLLVE.

Introduction

Low-light enhancement (LLE) is a classic computer vision challenge: how do you boost the signal and suppress the heavy noise (low SNR) found in dark environments? While deep learning has mastered this for static photos, applying these models to video usually results in "temporal instability"—the enhanced video jitters and flickers because the model treats every frame as an isolated event.

The obvious solution is to train on videos, but there is a massive data bottleneck. It is physically impossible to capture "ground truth" pairs of perfectly aligned low-light and normal-light videos of dynamic scenes at scale. Because of this, researchers usually settle for synthetic data or inconsistent image pairs, neither of which teaches a model how to stay stable over time.

Methodology

The authors propose a "Siamese" training strategy that uses a single image to "fake" a video sequence. Their pipeline consists of two main stages: motion prior inference and consistency-driven training.

The proposed Siamese training and testing framework.

Motion Prior Inference

Instead of applying simple global transformations (like rotating the whole image), the authors use instance segmentation (Detectron2) to identify objects and a Conditional Motion Propagation (CMP) network to predict plausible optical flow. This simulates how specific objects might move independently of the background.

Visualization of the motion prior inference process.

Consistency-Driven Training

The training loop takes a static image ($x_1$) and warps it using the predicted optical flow ($f$) to create a synthetic adjacent frame ($x_2$). Both frames are passed through the same enhancement network. The "Secret Sauce" is the Consistency Loss ($L_c$), which forces the network to ensure that the enhanced version of the warped frame ($g(x_2)$) matches the warped version of the enhanced first frame ($W(g(x_1), f)$). By penalizing the difference between these two paths, the network learns to be "transformation invariant," effectively smoothing out flickering artifacts.

Key Insights

  • Plausibility over Accuracy: The predicted optical flow doesn't need to be perfectly accurate. Even imperfect flow predictions act as helpful disturbances that challenge the network to remain stable, making the model more robust.
  • Local vs. Global Motion: Simple affine transforms aren't enough for complex scenes. By using instance-aware optical flow, the model learns to handle local object movements, which are the primary source of flickering in enhanced videos.
  • Zero Video Requirement: Temporal stability is a property of the model's behavior, not necessarily the training data's format. Video-like behavior can be baked into a model using a self-consistency loop on static images.

Experimental Results

The method was evaluated using the DAVIS Dataset, where high-quality frames were used as ground truth to synthesize a large-scale low-light dataset with realistic Gaussian and Poisson noise.

  • Quantitative Performance: On noisy synthetic data, the proposed method achieved a PSNR of 24.01 and SSIM of 0.9305, outperforming state-of-the-art image-based methods like SID.
  • Flicker Reduction: The method significantly reduced the Warping Error (WE)—a metric for temporal instability—achieving scores comparable to models trained on actual video sequences.
  • User Preference: In a blind study with 26 volunteers, 78.9% preferred the authors' results over other methods, citing better temporal stability and fewer visual artifacts.

Conclusion

This paper introduces a novel framework that enforces temporal consistency using only single images by simulating realistic motion through predicted optical flow. This allows models to achieve video-level stability without ever seeing a real training video. While the method relies on the quality of the initial motion prior, it is highly effective and generalizable. Future work aims to extend this approach to other restoration tasks like deraining and dehazing where paired video data is similarly scarce.

Summary updated

Abstract

Single image low light enhancement is an important task and it has many practical applications. Most existing methods adopt a single image approach. Although their performance is satisfying on a static single image, we found, however, they suffer serious temporal instability when handling low light videos. We notice the problem is because existing data-driven methods are trained from single image pairs where no temporal information is available. Unfortunately, training from real temporally consistent data is also problematic because it is impossible to collect pixel-wisely paired low and normal light videos under controlled environments in large scale and diversities with noise of identical statistics. In this paper, we propose a novel method to enforce the temporal stability in low light video enhancement with only static images. The key idea is to learn and infer motion field (optical flow) from a single image and synthesize short range video sequences. Our strategy is general and can extend to large scale datasets directly. Based on this idea, we propose our method which can infer motion prior for single image low light video enhancement and enforce temporal consistency. Rigorous experiments and user study demonstrate the state-of-the-art performance of our proposed method. Our code and model will be publicly available at https://github.com/zkawfanx/StableLLVE.
BibTeX
@inproceedings{zhang2021learning,
  author = {Fan Zhang and Yu Li and Shaodi You and Ying Fu},
  title = {{Learning Temporal Consistency for Low Light Video Enhancement From Single Images}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=vqx_5L_fJJ},
}
Figure