Abstract: This paper presents a general framework to build fast and accurate algorithms for video enhancement tasks such as super-resolution, deblurring, and denoising. Essential to our framework is the realization that the accuracy, rather than the density, of pixel flows is what is required for high-quality video enhancement. Most of prior works take the opposite approach: they estimate dense (per-pixel)—but generally less robust—flows, mostly using computationally costly algorithms. Instead, we propose a lightweight flow estimation algorithm; it fuses the sparse point cloud data and (even sparser and less reliable) IMU data available in modern autonomous agents to estimate the flow information. Building on top of the flow estimation, we demonstrate a general framework that integrates the flows in a plug-andplay fashion with different task-specific layers. Algorithms built in our framework achieve 1.78× — 187.41× speedup while providing a 0.42 dB – 6.70 dB quality improvement over competing methods.
Introduction
High-quality video enhancement tasks, such as super-resolution, deblurring, and denoising, rely heavily on "temporal alignment"—the ability to accurately track pixel movement across adjacent frames. Traditionally, this is achieved through dense optical flow estimation, which attempts to calculate a motion vector for every single pixel.
However, this approach faces a fundamental dilemma: to enhance a low-quality video (e.g., one that is blurry or noisy), you need accurate flow, but extracting accurate flow from degraded pixels is computationally expensive and often unreliable. This paper proposes a paradigm shift, prioritizing flow accuracy over density by leveraging sparse depth data from LiDAR and ego-motion from IMU sensors.
Proposed Method: LiDAR-Guided Flow Estimation
Instead of attempting to estimate motion from messy image content, the proposed framework calculates flow analytically using geometric projections.
Geometric Flow Calculation
By using precise 3D coordinates from LiDAR and the device's movement (ego-motion) from IMU sensors, the system can mathematically project where a point in a 3D scene will appear in subsequent camera frames. This calculation is purely geometric and independent of image quality, making it robust against noise and blur.
Addressing Data Sparsity
LiDAR data is significantly sparser than image data (e.g., 130k points vs. 1 million pixels). To bridge this gap, the authors employ two key strategies:
- Cloud Merging: Five consecutive point clouds are registered together using IMU data to create a denser 3D map.
- Blocked Warping: Assuming local motion consistency, the flow of a single LiDAR point is applied to a small neighborhood (e.g., $5 \times 5$ or $3 \times 3$ pixel blocks), effectively "thickening" the sparse flow field.
System Architecture
The framework follows a modular two-stage design. A common front-end handles temporal alignment by warping frames based on the calculated sparse flow. This aligned data is then passed to a task-specific back-end designed for the specific enhancement goal (super-resolution, deblurring, or denoising).
Experimental Evaluation
The framework was evaluated using the KITTI dataset, a standard benchmark for autonomous driving that provides synchronized video, LiDAR point clouds, and IMU data. The researchers simulated various degradations—low resolution, blur, and noise—to test the robustness of the enhancement algorithms.
Key Results
The proposed method, particularly the version using merged point clouds (OURS-M), demonstrated significant improvements in both speed and quality over state-of-the-art deep learning models:
- Quality Gains: The framework achieved a 1.83 dB PSNR improvement in deblurring over ESTRNN and a massive 6.70 dB improvement in denoising compared to DVDnet.
- Computational Efficiency: On an Nvidia RTX 2080, the denoising task was 187.4x faster than DVDnet. On mobile hardware (Jetson Xavier), the system maintained substantial speedups, such as being 7.24x faster than RBPN for super-resolution. The flow estimation itself is extremely lightweight, executing in only about 10 microseconds on a GPU.
Conclusion and Future Work
This research demonstrates that highly accurate sparse flow is more effective for video enhancement than dense but noisy estimated flow. While the current method requires specific hardware (LiDAR/IMU) and primarily accounts for camera ego-motion, it offers a powerful solution for autonomous agents like self-driving cars or AR glasses.
Future work will focus on better handling independently moving objects (like pedestrians) that may not be well-represented in sparse LiDAR data and optimizing the storage footprint of point cloud data.
Summary updated