Optimizing 4D Lookup Table for Low-light Video Enhancement via Wavelet Priori

Jinhong He, Minglong Xue, Wenhai Wang, Mingliang ZhouarXiv 2024 · DOI · arXiv

Abstract

Low-light video enhancement (LLVE) requires a delicate balance between visual quality, temporal stability, and computational efficiency. Traditional deep learning models often struggle with real-time constraints, while standard 3D Lookup Tables (LUTs) fail to capture the complex, non-linear color shifts inherent in dark environments. This paper introduces WaveLUT, a novel 4D LUT framework that integrates wavelet-based lighting priors and a multimodal text-driven training strategy. WaveLUT achieves state-of-the-art visual quality and temporal consistency while maintaining the extreme efficiency required for live applications, processing 1080p video at approximately 0.05 seconds per frame.

Introduction

Enhancing video captured in low-light conditions is a significant challenge. The primary goal is to improve visibility without introducing artifacts like flicker or color bias. While deep learning has advanced image quality, the computational overhead often precludes real-time video use. Conversely, Lookup Tables (LUTs) offer high speed but limited capacity; standard 3D LUTs cannot distinguish between different lighting contexts for the same input pixel value, leading to mapping errors.

WaveLUT addresses this "one-to-many" mapping problem by introducing a fourth dimension to the LUT structure, representing enhancement intensity. This allows the model to be context-aware, adapting its color transformation based on the surrounding environment.

Methodology

The WaveLUT architecture is built upon three core innovations designed to optimize color mapping for dynamic, low-light scenes.

The WaveLUT framework overview, showcasing the Wavelet-Prior 4D LUT generation, the dynamic fusion strategy, and the text-driven perceptual loss using CLIP.

1. Wavelet-Prior 4D LUT

Instead of a static 3D (RGB) table, WaveLUT utilizes a 4D structure where the fourth dimension is guided by a lighting prior. The system employs a Discrete Wavelet Transform (DWT) to extract low-frequency components from the input video. These components serve as a stable "rough lighting prior," separating illumination from high-frequency noise and detail.

2. Dynamic Fusion Strategy

Lighting is rarely uniform across a video frame. WaveLUT uses a spatial weighting mechanism to adaptively fuse the wavelet-derived lighting prior with the target intensity structure. This ensures the enhancement is localized and contextually relevant, preventing a "one-size-fits-all" filter effect that often plagues simpler models.

3. Text-Driven Appearance Reconstruction

To ensure perceptual quality, the model is trained using a multimodal approach. It leverages the CLIP (Contrastive Language-Image Pre-training) model to compare enhanced frames against semantic text prompts such as "high light image" and "clear image." This nudges the model toward results that are not just mathematically accurate, but visually pleasing to human observers.

Experimental Results

WaveLUT was evaluated on several prominent benchmarks, including SMID (still videos with long-exposure ground truths) and SDSD (dynamic indoor/outdoor subsets).

Performance comparison showing PSNR (dB) vs. Inference Time (s). WaveLUT (Ours) achieves the highest PSNR while maintaining competitive real-time inference speeds.

Quantitative and Qualitative Performance

  • Superior Accuracy: WaveLUT outperformed previous state-of-the-art methods like FASTLLVE, achieving a 0.57 dB PSNR improvement on the SDSD dataset and a 1.07 dB boost on SMID.
  • Real-Time Efficiency: As shown in the performance analysis, WaveLUT maintains a low inference time (approx. 0.05s per frame on an RTX 3090), making it suitable for 1080p live video processing.
  • Visual Fidelity: Qualitative assessments indicate significantly better color saturation and a marked reduction in ghosting or blurring artifacts compared to traditional deep-learning enhancers.

Conclusion and Future Work

WaveLUT demonstrates that 4D LUTs, when guided by wavelet priors and semantic text-driven loss, can bridge the gap between high-quality enhancement and real-time performance. However, the authors note that in extremely dark scenes with near-zero visual cues, constructing an accurate lighting prior remains challenging. Future research will focus on increasing robustness for these extreme edge cases and further optimizing the framework for deployment on lower-end mobile hardware.

Summary updated

Abstract

Low-light video enhancement is highly demanding in maintaining spatiotemporal color consistency. Therefore, improving the accuracy of color mapping and keeping the latency low is challenging. Based on this, we propose incorporating Wavelet-priori for 4D Lookup Table (WaveLUT), which effectively enhances the color coherence between video frames and the accuracy of color mapping while maintaining low latency. Specifically, we use the wavelet low-frequency domain to construct an optimized lookup prior and achieve an adaptive enhancement effect through a designed Wavelet-prior 4D lookup table. To effectively compensate the a priori loss in the low light region, we further explore a dynamic fusion strategy that adaptively determines the spatial weights based on the correlation between the wavelet lighting prior and the target intensity structure. In addition, during the training phase, we devise a text-driven appearance reconstruction method that dynamically balances brightness and content through multimodal semantics-driven Fourier spectra. Extensive experiments on a wide range of benchmark datasets show that this method effectively enhances the previous method's ability to perceive the color space and achieves metric-favorable and perceptually oriented real-time enhancement while maintaining high efficiency.
BibTeX
@misc{he2024optimizing,
  author = {Jinhong He and Minglong Xue and Wenhai Wang and Mingliang Zhou},
  title = {{Optimizing 4D Lookup Table for Low-light Video Enhancement via Wavelet Priori}},
  year = {2024},
  eprint = {2409.08585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.08585},
}
Figure