TL;DR
This paper introduces STLVQE, a real-time, low-latency framework designed for online video quality enhancement (Online-VQE). By sharing features uniformly across modules and replacing heavy deep neural network layers with novel Spatial-Temporal Look-up Tables (ST-LUTs), it achieves high-quality enhancement at over 40 FPS for 720p video without relying on future frames.
What Problem Does This Paper Address?
Real-time video applications like video conferencing, live webcasting, and cloud gaming suffer from visual artifacts due to aggressive video compression (e.g., H.264/AVC or H.265/HEVC). While offline Video Quality Enhancement (VQE) methods achieve state-of-the-art results by leveraging temporal information across multiple frames, they are fundamentally unsuited for online scenarios. They suffer from two fatal flaws:
- High Latency: They often require future frames to enhance the current frame, introducing uncontrollable streaming delays.
- Slow Inference: They rely on heavy, computationally expensive deep learning architectures that cannot process high-resolution video in real-time.
Existing low-latency methods either process frames individually (ignoring valuable temporal context) or remain too slow for practical deployment. The authors address this gap by proposing a framework that utilizes only past and current frames while maintaining real-time processing speeds.
Methods
The authors propose STLVQE, a lightweight framework consisting of three core components designed to eliminate computational redundancy:
- Module-Agnostic Feature Extractor (MAFE): Traditional VQE pipelines use separate feature extractors for frame propagation, alignment, and enhancement. MAFE extracts a single latent feature map once and shares it uniformly across all subsequent modules, significantly reducing redundant computations.
- The Propagation Module: To avoid complex reference-frame selection networks, the framework utilizes a Reference Frame Window (RFW) and selects past frames based on their compression Quantization Parameter (QP) value (smaller QP = higher quality). Extracted features are stored in a Temporal Cache (TC) to eliminate re-computation loops.
- The Alignment Module: A lightweight offset predictor replaces heavy optical flow or non-local networks to align reference frames to the current frame via deformable convolutions.
- Spatial-Temporal Look-up Tables (ST-LUTs): This is the core innovation. During training, a compact CNN learns spatial and temporal enhancements. For inference, this CNN is converted into look-up tables. Complex convolutional operations are replaced with direct, ultra-fast memory access queries. The system splits these into S-LUTs (for spatial context) and T-LUTs (for cross-frame temporal context).
Datasets
The model was trained and evaluated on the standard MFQE 2.0 dataset, which contains 126 video sequences across multiple resolutions (ranging from 240p to 1600p). Following standard protocols, 108 videos were used for training and 18 for testing. All videos were compressed using the HEVC test model (HM16.5) in Low Delay P (LDP) mode across five Quantization Parameters (QP = 22, 27, 32, 37, and 42).
Key Insights and Ideas
- LUTs for Time: While Look-up Tables have been successfully used to speed up single-image super-resolution, this work is the first to extend LUT structures to the temporal dimension for video tasks. By querying parallel tables with cross-frame pixel coordinates, they capture inter-frame correlations instantly.
- QP as a "Free" Quality Indicator: Instead of running an expensive neural network to figure out which past frames contain the best reference details, the framework simply reads the video bitstream's QP value—gaining structural awareness for zero computational cost.
- Ditching Redundancy: Standard VQE architectures sacrifice speed by re-extracting features within different sub-modules. Uniformly sharing a single latent feature map yields massive speedups without sacrificing performance.
Key Results
- Real-Time Speed: STLVQE achieves an average processing latency of 24.7 ms per frame on 720p video using an NVIDIA RTX 3090, translating to over 40 FPS. It is the only method in its performance class suitable for real-time streaming.
- Superior Trade-off: It outperforms all existing low-latency methods. Compared to the previous best low-latency method (DCAD), STLVQE improves $\Delta$PSNR by 0.17 dB and $\Delta$SSIM by 0.36 $\times 10^{-2}$, while running 9.6 ms faster per frame.
- Ultra-Lightweight: The model size is incredibly small at 398 KB, with the look-up tables occupying 228 MB of memory, easily fitting into low-end or edge GPUs.
Limitations and Future Work
- Receptive Field Constraints: Because LUT storage size scales exponentially with the number of input pixels (the indexing space), the underlying enhancement network's receptive field is strictly capped (typically to a neighborhood of 4 or 6 pixels). This prevents the model from capturing long-range spatial context.
- Quantization Dependency: The method relies heavily on Learnable Quantization (LSQ) to compress pixel values into a small bit-range for the LUT index space, which can introduce minor precision loss compared to full deep CNNs.
Summary updated