Deep Reparametrization of Multi-Frame Super-Resolution and Denoising

Goutam Bhat, Martin Danelljan, Fisher Yu, Luc Van Gool, Radu TimofteICCV 2021 · OpenReview

Abstract

We propose a deep reparametrization of the maximum a posteriori (MAP) formulation commonly employed in multi-frame image restoration tasks. Our approach is derived by introducing a learned error metric and a latent representation of the target image, which transforms the MAP objective to a deep feature space. The deep reparametrization allows us to directly model the image formation process in the latent space and to integrate learned image priors into the prediction. Our approach thereby leverages the advantages of deep learning while also benefiting from the principled multi-frame fusion provided by the classical MAP formulation. We validate our approach through comprehensive experiments on burst denoising and burst super-resolution datasets. Our approach sets a new state-of-the-art for both tasks, demonstrating the generality and effectiveness of the proposed formulation.

Introduction

Multi-frame image restoration (MFIR) is a critical task in modern computational photography, aiming to reconstruct a high-quality image from a "burst" of noisy, low-resolution, and slightly misaligned frames. Traditionally, this problem has been addressed using Maximum A Posteriori (MAP) estimation. This classical framework minimizes a reconstruction error—measuring how well the estimate matches the observed data—alongside a regularization term that acts as a prior on the structure of clean images.

Despite its mathematical rigor, classical MAP estimation faces two significant hurdles. First, the degradation model ($H$) must be manually defined, requiring precise knowledge of blur kernels and downsampling factors that are rarely known in real-world scenarios. Second, the image prior ($R$) must be hand-designed (e.g., Total Variation), which often results in over-smoothed or "cartoonish" artifacts. While modern deep learning methods have largely replaced these classical pipelines with black-box convolutional neural networks (CNNs), they often lack the explicit geometric and statistical fusion principles that make MAP estimation robust. This research introduces a "deep reparametrization" that bridges the gap between these two paradigms.

Methodology: Deep Reparametrization

The core innovation of this work is the transformation of the classical MAP objective into a deep latent space. This is achieved through three primary components:

1. Learned Error Metric (Encoder $E$)

Instead of calculating reconstruction errors in raw pixel space—which typically assumes simplistic Gaussian noise—the model passes input frames through a learned encoder. This maps the frames into a feature space where complex, non-linear noise distributions are "straightened out" into a representation that is more amenable to optimization.

2. Latent Representation (Decoder $D$)

Rather than optimizing the output image $y$ directly, the framework optimizes a latent representation $z$. The final restored image is then generated by a decoder ($y = D(z)$). This decoder effectively serves as a learned prior, implicitly capturing the complex textures and structural regularities of natural images better than any hand-crafted mathematical formula.

3. Feature Space Degradation ($G$)

The complex, image-space degradation $H$ is replaced by a learned operator $G$ in the feature space. By constraining $G$ to be a simple linear convolution, the authors force the encoder and decoder to learn a space where the underlying physics of the camera sensor and optics are simplified and linearized.

Optimization as a Layer

To solve the reparametrized objective, the authors utilize an unrolled differentiable optimizer. By implementing a steepest-descent algorithm as part of the neural network, the entire system—including the encoder, decoder, degradation operator, and a "certainty predictor"—can be trained end-to-end. The certainty predictor is particularly vital, as it allows the model to dynamically weight different frames, effectively ignoring regions that are poorly aligned or excessively noisy to prevent ghosting artifacts.

Experimental Results

The proposed method, referred to as MFIR, was evaluated across several rigorous benchmarks for both super-resolution and denoising.

Burst Super-Resolution

On the SyntheticBurst dataset, which uses an inverse camera pipeline to generate RAW bursts, the model outperformed the previous state-of-the-art (DBSR) by +0.80 dB PSNR. On the BurstSR dataset, which consists of real-world smartphone bursts paired with DSLR ground truth, the method maintained a significant lead of +0.28 dB.

Burst Denoising

The gains in denoising were even more pronounced. On grayscale and color burst denoising datasets derived from the Open Images collection, the model outperformed the Basis Prediction Network (BPN) by +1.19 dB on grayscale and a substantial +1.84 dB on color bursts.

Efficiency

Despite the iterative nature of the optimization, the model remains computationally efficient. By utilizing an initializer network to provide a high-quality starting point for the latent representation, the authors found that only three iterations of the optimizer were required to reach peak performance.

Conclusion and Future Work

This work demonstrates that the principled fusion of classical MAP estimation can be successfully integrated with the representational power of deep learning. By linearizing complex degradations through depth, the MFIR framework achieves superior results in both super-resolution and denoising tasks.

However, some challenges remain. The method currently relies on external motion estimation (such as PWC-Net), meaning the quality of restoration is still partially dependent on the accuracy of frame alignment. Furthermore, while the certainty predictor handles minor misalignments, complex dynamic scenes with fast-moving objects or occlusions remain a frontier for future research. The authors suggest that this deep reparametrization framework could be extended to other inverse problems, such as deblurring or medical imaging (MRI/CT), where the degradation processes are complex but the multi-shot principle remains applicable.

Summary updated

Abstract

We propose a deep reparametrization of the maximum a posteriori formulation commonly employed in multi-frame image restoration tasks. Our approach is derived by introducing a learned error metric and a latent representation of the target image, which transforms the MAP objective to a deep feature space. The deep reparametrization allows us to directly model the image formation process in the latent space, and to integrate learned image priors into the prediction. Our approach thereby leverages the advantages of deep learning, while also benefiting from the principled multi-frame fusion provided by the classical MAP formulation. We validate our approach through comprehensive experiments on burst denoising and burst super-resolution datasets. Our approach sets a new state-of-the-art for both tasks, demonstrating the generality and effectiveness of the proposed formulation.
BibTeX
@inproceedings{bhat2021deep,
  author = {Goutam Bhat and Martin Danelljan and Fisher Yu and Luc Van Gool and Radu Timofte},
  title = {{Deep Reparametrization of Multi-Frame Super-Resolution and Denoising}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=Ykcf0pZcnHp},
}
Figure