Seeing Dynamic Scene in the Dark: A High-Quality Video Dataset with Mechatronic Alignment

Ruixing Wang, Xiaogang Xu, Chi-Wing Fu, Jiangbo Lu, Bei Yu, Jiaya Jia

Introduction

Enhancing low-light video is a significant challenge in computer vision. When brightening underexposed frames, random sensor noise is inevitably amplified, leading to visual artifacts and temporal flickering. The ideal solution involves training deep learning models on "paired data"—perfectly aligned noisy and clean video sequences. However, capturing such data for dynamic scenes is notoriously difficult. Existing datasets often suffer from misalignment, low resolution, or a lack of motion.

This paper introduces SDSD (Seeing Dynamic Scenes in the Dark), a high-quality dataset of spatially and temporally aligned video pairs. To complement this dataset, the authors propose a novel deep learning framework that leverages self-supervised noise estimation and progressive alignment to restore dark videos without the typical "watercolor" artifacts of heavy denoising.

Mechatronic Alignment: A Hardware-First Approach

The core contribution of the SDSD dataset is its unprecedented alignment quality. Rather than relying on software-based optical flow—which often fails in extreme low-light conditions—the researchers utilized a mechatronic system.

The mechatronic system used for data collection, featuring a Canon 6D Mark II on a high-precision electric slide rail.

The setup consists of a professional DSLR mounted on a high-precision electric slide rail. By programming the rail to repeat the exact same motion path twice, the team could capture a "ground truth" version in normal light and a noisy, underexposed version (using ND filters and high ISO) of the same dynamic scene. With a mechanical error margin of less than 1mm, the resulting 150 video pairs (37,500 frames) provide a robust foundation for training.

The Enhancement Framework

The proposed architecture addresses the problem through three specialized modules:

  1. Progressive Alignment (PAL): Utilizing Deformable Convolutions (DCN), this module aligns neighboring frames to a central reference. This temporal integration allows the model to "borrow" details from adjacent frames, effectively reducing flickering.
  2. Self-Supervised Noise Estimation (SNE): To handle real-world sensor noise, the model employs a "Noisier2Noise" strategy. It learns to predict noise maps directly from the input data in a self-supervised manner, avoiding the pitfalls of synthetic noise models.
  3. Illumination Map Prediction (IMP): Based on Retinex theory, this module restores brightness and contrast independently of noise. By decoupling these processes, the framework avoids over-amplifying noise during the brightening phase.

Key Results and Insights

The SDSD framework was evaluated against several state-of-the-art (SOTA) baselines. Quantitatively, it achieved a PSNR of 24.92 dB, a significant improvement over previous methods.

Enhanced video frame using the proposed method, demonstrating high clarity and noise suppression.

Qualitatively, the results are marked by sharper details and natural color reproduction. A user study involving 100 participants confirmed these findings, with the proposed method being preferred in the majority of cases due to its lack of temporal flicker and superior photo-realism.

Robotics for Computer Vision

A key takeaway from this work is the effectiveness of using robotics to solve alignment problems. By ensuring identical camera trajectories mechanically, the researchers bypassed the mathematical complexities of post-hoc alignment, resulting in a dataset far cleaner than those relying on traditional software methods.

Conclusion and Future Work

While the mechatronic rail provides perfect alignment for linear motion, it does not yet account for the complex, non-linear "shake" of handheld devices. Future iterations may involve 6-degree-of-freedom robotic arms to better simulate real-world mobile cinematography. Nonetheless, SDSD represents a major step forward in high-quality low-light video restoration.

Summary updated

Abstract

Low-light video enhancement is an important task. Previous work is mostly trained on paired static images or videos. We compile a new dataset formed by our new strategy that contains high-quality spatially-aligned video pairs from dynamic scenes in low- and normal-light conditions. We built it using a mechatronic system to precisely control the dynamics during the video capture process, and further align the video pairs, both spatially and temporally, by identifying the system’s uniform motion stage. Besides the dataset, we propose an end-to-end framework, in which we design a self-supervised strategy to reduce noise, while enhancing the illumination based on the Retinex theory. Extensive experiments based on various metrics and large-scale user study demonstrate the value of our dataset and effectiveness of our method. The dataset and code are available at https://github.com/dvlab-research/SDSD.
BibTeX
@inproceedings{wang2021seeing,
  author = {Ruixing Wang and Xiaogang Xu and Chi-Wing Fu and Jiangbo Lu and Bei Yu and Jiaya Jia},
  title = {{Seeing Dynamic Scene in the Dark: A High-Quality Video Dataset with Mechatronic Alignment}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00956},
}
Figure