Seeing Dark Videos via Self-Learned Bottleneck Neural Representation

Haofeng Huang, Wenhan Yang, Lingyu Duan, Jiaying Liu

Low-light video enhancement is a notoriously difficult task in computer vision. Traditional supervised methods require massive, perfectly aligned datasets of "dark" and "bright" video pairs—a resource-intensive requirement that often leads to "domain gaps," where models trained on synthetic data fail to generalize to real-world conditions.

In their AAAI-24 paper, Haofeng Huang and colleagues from Peking University and Peng Cheng Laboratory propose SLBNR (Self-Learned Bottleneck Neural Representation). This approach requires zero external training data, learning to enhance a video by optimizing directly on the noisy, dark input sequence itself.

The Challenge of Low-Light Video

Capturing video in low light typically results in two major issues: extreme noise and "color flicker" caused by unstable white balance. While deep learning has advanced image restoration, applying it to dark videos remains tricky. Supervised methods often fail in authentic dark scenes due to training data limitations. Conversely, existing self-supervised methods often struggle with noise overfitting (e.g., Blind-Spot Networks) or introduce artifacts if frames are not perfectly aligned (e.g., Noise2Noise).

The Joint Bottleneck Framework

The core of the SLBNR solution is a Joint Bottleneck approach that constrains the learning process from two directions: the input content and the optimization objective.

Comparison of enhancement paradigms: (a) Input, (b) Supervised, (c) UDVD, (d) Frame2Frame, and (e) the proposed SLBNR with joint bottlenecks.
  1. Hybrid Neural Representation (Content Bottleneck): Instead of relying solely on pixel coordinates, the model uses a compact deep embedding. By restricting this embedding to a low-dimensional space, the network is forced to capture the consistent, intrinsic structure of the scene while ignoring random, high-frequency noise.
  2. Entropy Minimization (Objective Bottleneck): Noise is inherently chaotic and high-entropy. The authors model the signal distribution using a Gaussian-Mixture Model (GMM) and optimize the network to minimize entropy. This effectively "squeezes out" noise from the final prediction.

As shown in the figure above, neither a content bottleneck nor an objective bottleneck is sufficient on its own. Combining them allows SLBNR to integrate information across multiple frames naturally, leading to superior temporal consistency without heavy-duty alignment.

Chromatic Retinex Decomposition

To address the color flicker common in dark videos, the authors developed a novel decomposition method. Unlike standard Retinex models that assume monochromatic light, this version accounts for chromatic (colored) illumination. By isolating color fluctuations into a specific illumination layer, the model can maintain a temporally consistent "reflectance" layer (the actual objects). This decoupling makes self-supervised warping losses significantly more stable in low-light conditions.

Experimental Results

The authors evaluated SLBNR on the DRV (Dynamic Raw Video) dataset, using real-world dynamic videos of dark scenes. Since ground truth is unavailable for these sequences, the performance was measured using three standard no-reference metrics: NIQE, ILNIQE, and NIQMC.

SLBNR achieved the best scores across all metrics, outperforming unsupervised image methods (Zero-DCE), supervised video methods (SDSD), and unsupervised denoising methods (UDVD).

Method NIQE ↓ ILNIQE ↓ NIQMC ↑
Zero-DCE 5.1637 44.0923 3.9014
SDSD (Supervised) 13.0713 59.5753 3.9162
UDVD (Self-supervised) 13.5302 51.8735 2.6228
Ours (SLBNR) 4.7616 31.1489 4.8123

Qualitatively, SLBNR produces clear frames with natural colors and sharp details, whereas other methods often leave behind heavy noise or create over-smoothed, blurry images.

Conclusion

SLBNR demonstrates that high-quality video enhancement is possible without external training data. By leveraging a joint bottleneck strategy and chromatic Retinex decomposition, the framework effectively suppresses noise and corrects color flicker. While the current optimization process is slower than one-shot supervised models, the ability to generalize to real-world dark scenes makes it a powerful tool for authentic low-light restoration.

Summary updated

Abstract

Enhancing low-light videos in a supervised style presents a set of challenges, including limited data diversity, misalignment, and the domain gap introduced through the dataset construction pipeline. Our paper tackles these challenges by constructing a self-learned enhancement approach that gets rid of the reliance on any external training data. The challenge of self-supervised learning lies in fitting high-quality signal representations solely from input signals. Our work designs a bottleneck neural representation mechanism that extracts those signals. More in detail, we encode the frame-wise representation with a compact deep embedding and utilize a neural network to parameterize the video-level manifold consistently. Then, an entropy constraint is applied to the enhanced results based on the adjacent spatial-temporal context to filter out the degraded visual signals, e.g. noise and frame inconsistency. Last, a novel Chromatic Retinex decomposition is proposed to effectively align the reflectance distribution temporally. It benefits the entropy control on different components of each frame and facilitates noise-to-noise training, successfully suppressing the temporal flicker. Extensive experiments demonstrate the robustness and superior effectiveness of our proposed method. Our project is publicly available at: https://huangerbai.github.io/SLBNR/.
BibTeX
@inproceedings{huang2024seeing,
  author = {Haofeng Huang and Wenhan Yang and Lingyu Duan and Jiaying Liu},
  title = {{Seeing Dark Videos via Self-Learned Bottleneck Neural Representation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i3.28006},
}
Figure