TL;DR
This paper introduces ALANET, a novel neural network that jointly tackles video deblurring and frame interpolation to generate high-frame-rate, sharp videos. Unlike previous methods that assume all input frames are blurry, ALANET adaptively handles videos with mixed blur and sharp frames by using a unique latent attention mechanism.
What Problem Does This Paper Address?
Modern videos, especially those captured by mobile devices, often suffer from two main issues: motion blur and a low frame rate. Motion blur degrades image quality, while a low frame rate leads to jerky motion and a lack of temporal smoothness. Existing solutions typically address these problems separately (deblurring or interpolation) or make a strong assumption that all input frames are blurry. This is unrealistic, as real-world videos often contain a mix of blurry and sharp frames. The authors aim to solve the more challenging problem of generating high-frame-rate, sharp videos from low-frame-rate, potentially mixed-quality inputs, without prior knowledge of which frames are blurry.
Methods
ALANET is built upon a U-Net-like architecture, comprising three main components:
- Encoder ($\mathcal{E}$): This convolutional neural network projects input video frames into a latent space, creating feature representations for each frame.
- Adaptive Latent Attention Module ($\mathcal{M}$): This is the core innovation. It operates in the latent space and uses a combination of self-attention and cross-attention mechanisms.
- Self-attention helps the model focus on important features within a single frame's latent representation.
- Cross-attention allows the model to leverage information from neighboring frames' latent representations. Crucially, this module adaptively assigns importance: if an input frame is sharp, it relies less on neighboring frames; if it's blurry, it utilizes more information from neighbors to aid deblurring and interpolation.
- Generator ($\mathcal{G}$): This network takes the optimized latent representations produced by the attention module and synthesizes the final high-frame-rate, sharp video frames.
The key novelty lies in applying attention mechanisms in the latent space rather than the pixel space, and doing so adaptively based on frame quality.
Datasets
- Adobe240 Dataset: A benchmark dataset containing videos captured at 240 fps. The authors used 110 videos for training and 8 for evaluation, following a common split.
- YouTube240 Dataset: A dataset crawled from YouTube, also at 240 fps, used for testing. The model was trained on Adobe240 and then tested on this dataset without further fine-tuning.
For training and evaluation, low frame-rate (30fps) videos were generated from the original high-frame-rate videos by averaging 11 consecutive frames (simulating blur). All frames were resized to 640x352.
Key Insights and Ideas
- Latent Space Attention is Powerful: Applying attention mechanisms in the compressed latent space, rather than directly on pixels, is more efficient and effective for capturing complex temporal relationships and feature interactions.
- Adaptive Handling of Frame Quality: The ability to dynamically adjust reliance on neighboring frames based on the input frame's blurriness is critical for handling real-world videos where blur is not uniform. This adaptive nature is a significant improvement over methods assuming uniform blur.
- Joint Deblurring and Interpolation Synergy: By jointly optimizing for both tasks within the latent space, the model can better leverage information that benefits both deblurring (e.g., sharp features from neighbors) and interpolation (e.g., motion cues).
- Self-Attention for Intra-Frame Focus, Cross-Attention for Inter-Frame Context: The combination of self-attention (within a frame) and cross-attention (between frames) provides a comprehensive way to extract and utilize relevant information for generating high-quality frames.
Key Results
- Quantitative Performance: ALANET achieved state-of-the-art or competitive results on both Adobe240 and YouTube240 datasets for joint deblurring and interpolation.
- On Adobe240, it reported a relative improvement of 2.3% in average PSNR and 1.04% in SSIM index against the state-of-the-art BIN4 method for the joint task.
- For deblurring alone on Adobe240, it showed a 1.04dB PSNR improvement and 2.09% SSIM improvement over BIN4.
- For interpolation alone on Adobe240, ALANET achieved 32.98dB PSNR, compared to 32.51dB for BIN4.
- When tested on YouTube240 (trained on Adobe240), ALANET achieved an average PSNR of 35.89dB and SSIM of 0.9504 for the joint task.
- Qualitative Results: Visual comparisons (Figure 4 & 5) demonstrate ALANET's superior ability to reduce motion blur and generate sharper, more visually appealing interpolated frames compared to existing methods like BIN4. The ablation studies (Figure 5, Table 3) highlight that combining self-attention and cross-attention yields the best results, outperforming methods using only one type of attention.
Limitations and Future Work
- Limitations: The paper acknowledges that while their method doesn't assume uniform blur, it still generates low-frame-rate videos by averaging frames. The performance might be further impacted by extremely severe or complex blur patterns not fully captured by this averaging simulation.
- Future Work: The authors suggest that their approach of using adaptive attention in the latent space for joint deblurring and interpolation is a promising direction. Further exploration could involve more sophisticated latent space representations or attention mechanisms, and testing on a wider variety of real-world video degradation scenarios.
Summary updated