Abstract
Burst Image Super-Resolution (SR) aims to reconstruct a high-quality, high-resolution (HR) image from a sequence of low-resolution (LR) frames. A major challenge in this task is the precise alignment of frames captured with handheld devices, where natural tremors introduce varying shifts. While Kernel Prediction Networks (KPNs) have emerged as a robust alternative to explicit optical flow estimation, existing multi-kernel approaches often fail to distinguish between the importance of different kernel sizes. This paper presents the Weighted Multi-Kernel Prediction Network (WMKPN), which introduces a dynamic weighting mechanism to adaptively fuse kernels of various sizes at each pixel. By learning these interdependencies, WMKPN achieves superior alignment and reconstruction, outperforming state-of-the-art methods on both synthetic and real-world datasets.
Problem Statement: The Alignment Challenge
Burst Image Super-Resolution involves combining a sequence of low-resolution images—typically captured by a handheld smartphone—into a single high-quality HR image. The primary hurdle is alignment. Hand tremors cause each frame in a burst to be slightly shifted. Traditional optical flow methods estimate motion explicitly but are prone to artifacts if the estimation is inaccurate.
To address this, researchers developed Kernel Prediction Networks (KPNs). Instead of warping pixels, KPNs predict a local filter (kernel) for each pixel that implicitly "gathers" information from neighboring frames. However, existing multi-kernel approaches (MKPN) treat kernels of different sizes (e.g., $3\times3$ vs $7\times7$) with equal importance. In practice, a pixel in a flat area requires a different receptive field than one on a sharp edge. WMKPN addresses this lack of discriminative learning between varying kernel sizes.
Methodology
The authors propose a dual-branch architecture based on a modified U-Net:
- Multi-Kernel Prediction Branch: This branch predicts values for multiple sets of kernels with different sizes ($S \in \{1, 3, 5, 7\}$). To maintain computational efficiency, the model uses separable kernels, predicting two 1D kernels ($1 \times s$ and $s \times 1$) instead of one 2D kernel.
- Kernel Weight Branch: This branch predicts a weight for each kernel size at every pixel location, normalized using a softmax function.
- Weighted Fusion: The final alignment kernel is a weighted sum of the kernels from the first branch, allowing the network to "choose" the most reliable kernel size for specific image regions.
- SR Reconstruction: After alignment, features are concatenated and passed through Residual Blocks (RBs) and a sub-pixel convolution layer to produce the final HR output.
Experimental Results
The researchers evaluated WMKPN on the synthetic Zurich RAW to RGB dataset and the real-world BurstSR dataset.
Quantitative Performance
On the real-world BurstSR dataset, WMKPN achieved a PSNR of 41.87 dB, outperforming the standard MKPN (41.69 dB) and other sophisticated architectures like RAMS (41.57 dB).
Qualitative Analysis
Visual comparisons demonstrate that WMKPN is significantly better at restoring high-frequency details, such as fine textures on building facades, which often appear blurry in other models.
As shown in the figure above, the weighting mechanism allows the network to adapt its receptive field to the local structure, resulting in a reconstruction that closely matches the ground truth (GT).
Conclusion
WMKPN demonstrates that kernel sizes should not be treated as independent tools. By learning their interdependencies via a weighting branch, the model can emphasize larger kernels for areas with more motion and smaller kernels for fine, static textures. This approach avoids the warping artifacts common in optical flow methods and provides a robust solution for real-world smartphone photography. Future work may explore applying this weighted multi-kernel approach to video deblurring and denoising.
Summary updated