Burst Image Restoration and Enhancement

Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Shahbaz Khan, Ming-Hsuan Yang

TL;DR

Modern smartphone cameras capture bursts of images to improve quality, but these bursts often suffer from misalignment and multiple degradations. This paper introduces Burstormer, a novel transformer-based architecture that effectively aligns and fuses these burst frames to produce superior single images. Burstormer achieves state-of-the-art results in super-resolution, denoising, and low-light enhancement while being computationally efficient.

What Problem Does This Paper Address?

Smartphone cameras, despite rapid advancements, still face limitations in hardware size, leading to compromises in image quality. To overcome this, computational photography techniques leverage bursts of images captured in quick succession. The core challenge lies in effectively merging the complementary information from these multiple frames, which are often misaligned due to motion and suffer from various degradations like noise and low dynamic range. Existing methods struggle with accurately aligning these frames, especially in the presence of complex motion, leading to artifacts. This paper aims to address this by proposing a robust and efficient method for aligning and fusing burst frames to reconstruct high-quality images.

Methods

The proposed Burstormer architecture is a novel transformer-based framework designed for burst image processing. It consists of two main parts:

  • Enhanced Deformable Alignment (EDA): This is a multi-scale, hierarchical module that tackles misalignment.
  • Burst Feature Attention (BFA): First, it extracts noise-free features using a transformer block that captures both local and non-local context.
  • Feature Alignment (FA): It then uses a modulated deformable convolution to align features from each burst frame to a reference frame. This alignment is performed hierarchically, starting from lower resolutions to handle sub-pixel shifts more effectively.
  • Reference-Based Feature Enrichment (RBFE): To further refine alignment and consolidate features, RBFE allows additional interaction between aligned frame features and the reference frame features. This module utilizes a Burst Feature Fusion (BFF) unit, which combines back-projection and squeeze-excitation mechanisms.
  • Image Reconstruction: This stage takes the aligned burst features and reconstructs the final high-quality image.
  • Adaptive Burst Pooling (ABFP): This module normalizes the input burst size to a constant number of frames (8 in this case) for consistent processing.
  • No-Reference Feature Enrichment (NRFE): This module progressively aggregates and upsamples the burst features. It employs a Cyclic Burst Sampling (CBS) mechanism to gather neighborhood frames in a zigzag manner, enabling long-range inter-frame communication efficiently. The fused features are then upsampled using pixel-shuffle.

Datasets

The authors evaluated Burstormer on several standard datasets for different tasks:

  • Burst Super-Resolution:
    • SyntheticBurst [3]: Contains synthetic RAW burst sequences.
  • BurstSR [3]: Contains real-world RAW burst sequences captured with smartphone cameras, with corresponding HR ground-truth images from DSLR cameras.
  • Burst Low-light Image Enhancement:
    • SID dataset [6] (Sony subset): Contains low-light RAW burst sequences and their corresponding well-exposed sRGB ground-truth images.
  • Burst Denoising:
    • Grayscale and Color Datasets [30, 41]: Standard datasets for evaluating denoising performance.

Key Insights and Ideas

  • Multi-scale Hierarchical Alignment: The EDA module's hierarchical approach, starting alignment from lower resolutions, is crucial for effectively handling sub-pixel misalignments, especially in the presence of complex motion.
  • Reference-Based Feature Enrichment: Explicitly allowing aligned features to interact with the reference frame (RBFE) significantly improves the refinement and consolidation of features, addressing residual misalignment issues.
  • Cyclic Burst Sampling for Efficient Communication: The CBS mechanism in NRFE enables effective long-range inter-frame communication without the computational overhead of traditional pseudo-burst generation, making the approach more efficient.
  • Unified Framework: Burstormer is presented as a unified model capable of handling multiple burst image processing tasks (super-resolution, denoising, low-light enhancement) with minimal modifications, showcasing its versatility.

Key Results

Burstormer achieved state-of-the-art performance across various tasks:

  • Burst Super-Resolution:
    • On the SyntheticBurst dataset, Burstormer achieved a PSNR of 42.83 dB, outperforming the previous best (BIPNet) by 0.9 dB, while using 47% fewer parameters and running 2x faster.
  • On the real BurstSR dataset, it achieved a PSNR of 48.82 dB, a 0.33 dB gain over BIPNet.
  • Qualitative results show significantly sharper details and fewer artifacts compared to competing methods.
  • Burst Low-light Image Enhancement:
    • Achieved a PSNR of 33.34 dB on the SID dataset, outperforming BIPNet by 0.47 dB and LEED by 3.54 dB.
  • Visual comparisons demonstrate more faithful color and structural details.
  • Burst Denoising:
    • Achieved a PSNR of 35.11 dB on grayscale denoising, a 0.75 dB improvement over BIPNet.
  • On color denoising, it showed similar improvements, with a 0.57 dB boost over BIPNet on high noise levels.
  • Denoised outputs were cleaner, sharper, and preserved subtle textures better.

Limitations and Future Work

  • Limitations: The paper does not explicitly detail limitations. However, the reliance on a reference frame for alignment might be a potential bottleneck in scenarios where the reference frame itself is severely degraded.
  • Future Work: The authors suggest that their framework is applicable to other burst processing tasks. The paper also mentions that their method is compute-efficient, implying potential for real-time applications on mobile devices. Further exploration of adaptive burst sampling strategies and more complex motion scenarios could be future directions.

Summary updated

Abstract

Burst Image Restoration aims to reconstruct a high-quality image by efficiently combining complementary inter-frame information. However, it is quite challenging since individual burst images often have inter-frame misalignments that usually lead to ghosting and zipper artifacts. To mitigate this, we develop a novel approach for burst image processing named BIPNet that focuses solely on the information exchange between burst frames and filter-out the inherent degradations while preserving and enhancing the actual scene details. Our central idea is to generate a set of pseudo-burst features that combine complementary information from all the burst frames to exchange information seamlessly. However, due to inter-frame misalignment, the information cannot be effectively combined in pseudo-burst. Thus, we initially align the incoming burst features regarding the reference frame using the proposed edge-boosting feature alignment. Lastly, we progressively upscale the pseudo-burst features in multiple stages while adaptively combining the complementary information. Unlike the existing works, that usually deploy single-stage up-sampling with a late fusion scheme, we first deploy a pseudo-burst mechanism followed by the adaptive-progressive feature up-sampling. The proposed BIPNet significantly outperforms the existing methods on burst super-resolution, low-light image enhancement, low-light image super-resolution, and denoising tasks.
BibTeX
@inproceedings{dudhane2021burst,
  author = {Akshay Dudhane and Syed Waqas Zamir and Salman Khan and Fahad Shahbaz Khan and Ming-Hsuan Yang},
  title = {{Burst Image Restoration and Enhancement}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr52688.2022.00567},
}
Figure