TL;DR
This paper introduces RVDD, a novel recurrent neural network architecture for joint denoising and demosaicing of raw video frames. By treating these traditionally separate steps as a unified task and leveraging temporal information through recurrence and motion compensation, RVDD achieves state-of-the-art results, setting a new baseline for video joint denoising and demosaicing.
What Problem Does This Paper Address?
Modern digital cameras capture raw sensor data, which is then processed by an Image Signal Processor (ISP) to produce the final sRGB image. Two crucial steps in this pipeline are denoising (removing sensor noise) and demosaicing (reconstructing full-color pixels from the Bayer pattern). Historically, these have been handled separately, often with denoising applied before demosaicing. However, recent research has shown benefits in jointly addressing these tasks, particularly for single images or bursts of frames. This paper tackles the less-explored problem of video joint denoising and demosaicing (JDD). Unlike single images or bursts, video processing demands temporal consistency and efficient processing of sequential frames, making existing single-image or burst JDD methods insufficient. The authors aim to establish a strong neural network baseline for this critical video processing task.
Methods
The core of the paper is the proposed Recurrent Video Joint Denoising and Demosaicing (RVDD) network. RVDD builds upon the U-Net architecture, a common choice for image restoration tasks. The key innovations lie in how it handles temporal information:
- Recurrence: RVDD employs recurrent connections, allowing information from previous frames (both the denoised output and intermediate features) to influence the processing of the current frame. This is crucial for maintaining temporal consistency in videos.
- Motion Compensation: To effectively utilize information from previous frames, RVDD incorporates explicit motion compensation. It uses optical flow to warp previous frames and their corresponding features to align them with the current frame before feeding them into the network.
- Feature Propagation: The network propagates not only the denoised RGB output from the previous frame but also its internal feature maps, providing a richer context for current frame processing.
- Future Frame Utilization: The best-performing variant of RVDD also incorporates information from the next frame, further enhancing temporal context.
The authors also explore a modified version of the FastDVDnet, a state-of-the-art video denoising network, adapted for JDD. They compare various configurations of their RVDD network, including recurrent vs. non-recurrent approaches, the inclusion of motion compensation, and different ways of propagating information.
Datasets
The authors introduce a simulated raw-to-sRGB realistic dataset for training and quantitative evaluation. This dataset is derived from the REDS dataset [43] and is tailored to mimic the characteristics of the CRVD dataset [64], a public dataset of real raw videos. They provide two versions of their synthetic dataset: one with stabilized motion and another with natural, unstabilized motion. This allows for evaluating the generalization capabilities of the networks across different motion patterns. The CRVD dataset itself is used for qualitative evaluation on real-world data.
Key Insights and Ideas
- Joint processing is superior: The paper strongly demonstrates that jointly denoising and demosaicing raw video frames leads to significantly better results than processing them separately (e.g., raw denoising followed by demosaicing).
- Recurrence is key for video: For video JDD, recurrent networks that leverage temporal information are essential and outperform non-recurrent approaches.
- Motion compensation is vital: Explicitly incorporating motion compensation, especially when dealing with unstabilized video, provides a substantial performance boost.
- RGB domain processing benefits raw quality: Even though the JDD network operates in the RGB domain, it surprisingly leads to improved PSNR in the raw domain compared to raw denoising networks. This suggests that end-to-end training in the RGB space allows the network to learn better representations that benefit both tasks.
- Future frame information helps: Utilizing information from the next frame, in addition to previous frames and features, further improves the quality of the denoised and demosaiced output.
Key Results
- The proposed RVDD network, particularly the recurrent version with motion compensation and future frame utilization, achieves the best performance across various metrics (PSNR and SSIM).
- RVDD outperforms a baseline raw denoiser followed by a demosaicing network by a significant margin (up to 2.2dB in PSNR).
- The recurrent RVDD network with motion compensation achieves better performance than non-recurrent VDD networks, even when the latter use more input frames.
- The generalization gap between stabilized and non-stabilized datasets is smaller for networks with motion compensation, highlighting its role in disentangling motion from image content.
- The ConvNeXt U-Net architecture shows faster convergence and slightly improved performance compared to the standard U-Net for the RVDD network.
Limitations and Future Work
- Dependence on Optical Flow: A primary limitation acknowledged by the authors is the network's reliance on external optical flow estimation. This adds computational overhead and can be a bottleneck. Future work will focus on improving this aspect, potentially by making the network end-to-end trainable without explicit optical flow.
- Generalization to Unseen Motion: While the paper evaluates generalization across stabilized and unstabilized datasets, further work could explore generalization to more diverse and complex motion patterns.
- Computational Efficiency: While RVDD sets a new performance baseline, further research could focus on optimizing its computational efficiency for real-time applications in mobile ISPs.
Summary updated