Abstract: To the best of our knowledge, the existing deep-learning-based Video Super-Resolution (VSR) methods exclusively make use of videos produced by the Image Signal Processor (ISP) of the camera system as inputs. Such methods are 1) inherently suboptimal due to information loss incurred by non-invertible operations in ISP, and 2) inconsistent with the real imaging pipeline where VSR in fact serves as a pre-processing unit of ISP. To address this issue, we propose a new VSR method that can directly exploit camera sensor data, accompanied by a carefully built Raw Video Dataset (RawVD) for training, validation, and testing. This method consists of a Successive Deep Inference (SDI) module and a reconstruction module, among others. The SDI module is designed according to the architectural principle suggested by a canonical decomposition result for Hidden Markov Model (HMM) inference; it estimates the target high-resolution frame by repeatedly performing pairwise feature fusion using deformable convolutions. The reconstruction module, built with elaborately designed Attention-based Residual Dense Blocks (ARDBs), serves the purpose of 1) refining the fused feature and 2) learning the color information needed to generate a spatial-specific transformation for accurate color correction. Extensive experiments demonstrate that owing to the informativeness of the camera raw data, the effectiveness of the network architecture, and the separation of super-resolution and color correction processes, the proposed method achieves superior VSR results compared to the state-of-the-art and can be adapted to any specific camera-ISP.
The Information Bottleneck in Traditional VSR
Traditional Video Super-Resolution (VSR) methods typically act as a post-processing step, operating on video that has already passed through a camera's Image Signal Processor (ISP). The ISP performs several non-invertible operations—such as demosaicing, denoising, sharpening, and compression—which act as information bottlenecks. By the time a VSR algorithm receives these frames, the bit-depth has often dropped from 14 bits to 8 bits, and fine textures have been blurred or replaced by compression artifacts.
This paper argues that VSR should instead be a pre-processing unit within the ISP, working directly with the "primitive radiance information" found in raw sensor data. By bypassing the ISP's destructive operations, VSR can leverage the full 12-14 bits of linear radiance information to produce significantly sharper and more accurate upscaled results.
Methodology: RawVSR and HMM Inference
The authors introduce RawVSR, a novel architecture built on two primary innovations:
- Successive Deep Inference (SDI) Module: Moving away from heuristic fusion rules, the SDI module is derived from Hidden Markov Model (HMM) inference principles. By treating high-resolution frames as hidden states and low-resolution raw frames as observations, the authors implement an iterative pairwise feature fusion approach using Deformable Convolutions for alignment. This mathematical grounding ensures the network effectively captures temporal correlations across multiple frames.
- Dual-Branch Reconstruction: The network separates the restoration process into two paths: a Texture Restoration Branch for high-frequency details and a Color Correction Branch that learns a spatial-specific transformation matrix to map raw data into standard color spaces (like sRGB). This disentanglement allows the model to be device-independent; users can change the color reference to match different camera "looks" without retraining the super-resolution engine.
The Raw Video Dataset (RawVD)
Because no public raw video datasets for VSR existed, the authors created RawVD. This dataset contains 110 videos (1920x1080) covering diverse scenes, captured using a Canon 5D3 in Magic Lantern Video (MLV) format. It provides the necessary HR/LR raw pairs and their processed RGB counterparts for rigorous training and benchmarking.
Key Results and Generalization
RawVSR significantly outperforms existing state-of-the-art (SOTA) methods that rely on processed RGB data. On the RawVD 4× VSR task, RawVSR achieved an average PSNR of 28.76 dB, outperforming the best processed-data method (EDVR) by nearly 1 dB.
Visual comparisons highlight the "Raw" advantage: text and fine textures that are blurred or lost in standard ISP-processed video are reconstructed with high clarity. Furthermore, despite being trained on Canon data, the model generalizes well to other hardware, such as iPhone 8p raw data, proving the robustness of its dual-branch architecture.
Conclusion
By exploiting camera raw data and utilizing a mathematically grounded HMM inference framework, RawVSR sets a new benchmark for video super-resolution. The separation of texture restoration from color correction not only improves performance but also offers a flexible, hardware-portable solution that aligns with the real-world imaging pipeline.
Summary updated