Online Video Quality Enhancement with Spatial-Temporal Look-up Tables

Zefan Qu, Xinyang Jiang, Yifan Yang, Dongsheng Li, Cairong Zhao

TL;DR

This paper introduces STLVQE, a real-time, low-latency framework designed for online video quality enhancement (Online-VQE). By sharing features uniformly across modules and replacing heavy deep neural network layers with novel Spatial-Temporal Look-up Tables (ST-LUTs), it achieves high-quality enhancement at over 40 FPS for 720p video without relying on future frames.

What Problem Does This Paper Address?

Real-time video applications like video conferencing, live webcasting, and cloud gaming suffer from visual artifacts due to aggressive video compression (e.g., H.264/AVC or H.265/HEVC). While offline Video Quality Enhancement (VQE) methods achieve state-of-the-art results by leveraging temporal information across multiple frames, they are fundamentally unsuited for online scenarios. They suffer from two fatal flaws:

  1. High Latency: They often require future frames to enhance the current frame, introducing uncontrollable streaming delays.
  2. Slow Inference: They rely on heavy, computationally expensive deep learning architectures that cannot process high-resolution video in real-time.

Existing low-latency methods either process frames individually (ignoring valuable temporal context) or remain too slow for practical deployment. The authors address this gap by proposing a framework that utilizes only past and current frames while maintaining real-time processing speeds.

Methods

The authors propose STLVQE, a lightweight framework consisting of three core components designed to eliminate computational redundancy:

Fig. 2: The framework of STLVQE (inference phase), which consists of a Module- Agnostic Feature Extractor and three main parts: the propagation, alignment and enhancement module. In the inference phase, the propagation module selects the ref- erence frame and accesses the relevant information, which is sent to the alignment module for temporal alignment, and finally the aligned reference frames and enhancing frame are input to the enhancement module for the final results.
  1. Module-Agnostic Feature Extractor (MAFE): Traditional VQE pipelines use separate feature extractors for frame propagation, alignment, and enhancement. MAFE extracts a single latent feature map once and shares it uniformly across all subsequent modules, significantly reducing redundant computations.
  2. The Propagation Module: To avoid complex reference-frame selection networks, the framework utilizes a Reference Frame Window (RFW) and selects past frames based on their compression Quantization Parameter (QP) value (smaller QP = higher quality). Extracted features are stored in a Temporal Cache (TC) to eliminate re-computation loops.
  3. The Alignment Module: A lightweight offset predictor replaces heavy optical flow or non-local networks to align reference frames to the current frame via deformable convolutions.
  4. Spatial-Temporal Look-up Tables (ST-LUTs): This is the core innovation. During training, a compact CNN learns spatial and temporal enhancements. For inference, this CNN is converted into look-up tables. Complex convolutional operations are replaced with direct, ultra-fast memory access queries. The system splits these into S-LUTs (for spatial context) and T-LUTs (for cross-frame temporal context).
Fig. 4: The specific schematic of our Spatial-Temporal Look-up Tables, which consists of two parts: Temporal-LUT (left side) and Spatial-LUT (right side).

Datasets

The model was trained and evaluated on the standard MFQE 2.0 dataset, which contains 126 video sequences across multiple resolutions (ranging from 240p to 1600p). Following standard protocols, 108 videos were used for training and 18 for testing. All videos were compressed using the HEVC test model (HM16.5) in Low Delay P (LDP) mode across five Quantization Parameters (QP = 22, 27, 32, 37, and 42).

Key Insights and Ideas

  • LUTs for Time: While Look-up Tables have been successfully used to speed up single-image super-resolution, this work is the first to extend LUT structures to the temporal dimension for video tasks. By querying parallel tables with cross-frame pixel coordinates, they capture inter-frame correlations instantly.
  • QP as a "Free" Quality Indicator: Instead of running an expensive neural network to figure out which past frames contain the best reference details, the framework simply reads the video bitstream's QP value—gaining structural awareness for zero computational cost.
  • Ditching Redundancy: Standard VQE architectures sacrifice speed by re-extracting features within different sub-modules. Uniformly sharing a single latent feature map yields massive speedups without sacrificing performance.

Key Results

  • Real-Time Speed: STLVQE achieves an average processing latency of 24.7 ms per frame on 720p video using an NVIDIA RTX 3090, translating to over 40 FPS. It is the only method in its performance class suitable for real-time streaming.
  • Superior Trade-off: It outperforms all existing low-latency methods. Compared to the previous best low-latency method (DCAD), STLVQE improves $\Delta$PSNR by 0.17 dB and $\Delta$SSIM by 0.36 $\times 10^{-2}$, while running 9.6 ms faster per frame.
Fig. 1: Comparison of ∆PSNR/∆SSIM, average runtime per frame(720P video) and parameters on MFQE 2.0 test set at QP=37. Our STLVQE method achieves a great trade-off in enhancement performance and inference speed.
  • Ultra-Lightweight: The model size is incredibly small at 398 KB, with the look-up tables occupying 228 MB of memory, easily fitting into low-end or edge GPUs.

Limitations and Future Work

  • Receptive Field Constraints: Because LUT storage size scales exponentially with the number of input pixels (the indexing space), the underlying enhancement network's receptive field is strictly capped (typically to a neighborhood of 4 or 6 pixels). This prevents the model from capturing long-range spatial context.
  • Quantization Dependency: The method relies heavily on Learnable Quantization (LSQ) to compress pixel values into a small bit-range for the LUT index space, which can introduce minor precision loss compared to full deep CNNs.

Summary updated

Abstract

Low latency rates are crucial for online video-based applications, such as video conferencing and cloud gaming, which make improving video quality in online scenarios increasingly important. However, existing quality enhancement methods are limited by slow inference speed and the requirement for temporal information contained in future frames, making it challenging to deploy them directly in online tasks. In this paper, we propose a novel method, STLVQE, specifically designed to address the rarely studied online video quality enhancement (Online-VQE) problem. Our STLVQE designs a new VQE framework which contains a Module-Agnostic Feature Extractor that greatly reduces the redundant computations and redesign the propagation, alignment, and enhancement module of the network. A Spatial-Temporal Look-up Tables (STL) is proposed, which extracts spatial-temporal information in videos while saving substantial inference time. To the best of our knowledge, we are the first to exploit the LUT structure to extract temporal information in video tasks. Extensive experiments on the MFQE 2.0 dataset demonstrate that our STLVQE achieves a satisfactory performance-speed trade-off.
BibTeX
@inproceedings{qu2024online,
  author = {Zefan Qu and Xinyang Jiang and Yifan Yang and Dongsheng Li and Cairong Zhao},
  title = {{Online Video Quality Enhancement with Spatial-Temporal Look-up Tables}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73220-1\_26},
}
Figure