On Measuring and Controlling the Spectral Bias of the Deep Image Prior

Zenglin Shi, Pascal Mettes, Subhransu Maji, Cees G. M. Snoek

TL;DR

This paper tackles two key limitations of the Deep Image Prior (DIP) for inverse imaging tasks: the lack of control over its learned prior and the need for an early stopping criterion to prevent performance degradation. The authors introduce a novel "Frequency-Band Correspondence" metric to quantify DIP's spectral bias (favoring low frequencies) and propose Lipschitz-controlled convolutions and Gaussian upsampling to manage this bias, leading to stable performance and an automatic stopping criterion.

What Problem Does This Paper Address?

The Deep Image Prior (DIP) is a powerful technique that leverages randomly initialized neural networks to solve inverse imaging problems (like denoising, deblurring, or super-resolution) without requiring paired training data. The network's parameters are optimized to reconstruct a single degraded image. However, DIP faces two significant challenges:

  1. Lack of Control: It's unclear how to influence the learned image prior beyond simply choosing a network architecture. This makes it difficult to guide the reconstruction process.
  2. Performance Degradation & Early Stopping: As the network optimizes, its performance initially improves but then degrades after reaching a peak. This necessitates an "oracle" stopping criterion to halt training at the optimal point, which is often impractical.

This paper aims to address these limitations by understanding and controlling the underlying mechanisms of DIP, specifically its "spectral bias."

Methods

The core of this work lies in understanding and manipulating the spectral bias of DIP.

  • Frequency-Band Correspondence (FBC) Metric: The authors introduce a novel metric to quantify DIP's spectral bias. It measures how well the Fourier spectrum of the network's output at each optimization step corresponds to the target image's Fourier spectrum, broken down into different frequency bands. This reveals that DIP learns low-frequency components much faster and better than high-frequency ones.
  • Lipschitz-Controlled Convolution Layer: To control the network's ability to fit high-frequency noise (which causes performance degradation), they propose a modified convolution layer. By enforcing Lipschitz continuity on the layer's weights, they can bound the magnitude of its Fourier coefficients, effectively limiting its capacity to learn high frequencies. This is achieved by normalizing the spectral norm of the weight matrix.
  • Gaussian-Controlled Upsampling Layer: Upsampling layers, common in DIP architectures, can introduce their own spectral bias, often favoring low frequencies and slowing convergence. The authors propose a Gaussian-controlled upsampling method that allows tuning the amount of smoothing. This balances the learning of low and high frequencies, improving convergence and performance.
  • Automatic Stopping Criterion: Based on the controlled spectral bias, the authors propose a criterion to automatically stop optimization. It monitors the ratio of blurriness to sharpness in the output image, which stabilizes once the spectral bias is controlled and performance plateaus, avoiding unnecessary computation.

Datasets

The paper evaluates its methods on standard datasets for various inverse imaging tasks:

  • Image Denoising: Dabov et al. (2007) dataset (9 RGB images) and CBSD68 (Roth and Black, 2009) dataset (68 RGB images).
  • JPEG Image Deblocking: Classic5 dataset (Foi et al., 2006) and LIVE1 dataset (Sheikh et al., 2006).
  • Image Inpainting: Standard dataset by Heide et al. (2015) (11 grayscale images) and CBSD68 dataset.
  • Super-resolution: Set14 and Set5 datasets.
  • Image Enhancement: Not explicitly named, but evaluated on various images.

No new datasets were introduced.

Key Insights and Ideas

  • Spectral Bias is Key: The paper strongly suggests that the spectral bias of DIP (learning low frequencies faster) is not just an artifact but a fundamental mechanism enabling it to learn image priors without explicit training data.
  • High-Frequency Noise is the Culprit: Performance degradation in DIP is primarily attributed to the network eventually fitting high-frequency noise, which can corrupt learned image details.
  • Controllable Layers for Controllable Priors: By introducing Lipschitz-controlled convolutions and Gaussian upsampling, the authors demonstrate that the "prior" learned by DIP can be actively controlled, not just passively accepted.
  • Upsampling's Hidden Bias: The paper highlights that even standard upsampling layers can introduce undesirable spectral biases that hinder convergence, and their proposed method offers a way to mitigate this.
  • Stopping Criterion as a Consequence: The automatic stopping criterion isn't an arbitrary addition but a natural outcome of successfully controlling the spectral bias, leading to stable optimization curves.

Key Results

  • Elimination of Performance Degradation: The proposed methods (Lipschitz normalization and Gaussian upsampling) prevent the typical performance decay observed in standard DIP over iterations.
  • Effective Automatic Stopping: The proposed stopping criterion allows for early termination of optimization with minimal loss in performance, saving computation.
  • Competitive Performance: The method achieves favorable results across various tasks (denoising, deblocking, inpainting, super-resolution, enhancement), often outperforming existing DIP variants and sometimes matching or exceeding supervised methods. For instance, in denoising on CBSD68, their method achieves better PSNR than DIP variants and is competitive with supervised methods.
  • Improved Detail Recovery: Qualitative results show better preservation of high-frequency details and fewer artifacts compared to previous DIP methods.

Limitations and Future Work

  • Parameter Tuning: While the proposed methods offer control, finding the optimal hyperparameters (e.g., λ for Lipschitz normalization, σ for Gaussian upsampling) for each specific task and image can still require empirical tuning.
  • Generalizability of Stopping Criterion: The paper demonstrates the stopping criterion's effectiveness across several tasks, but its robustness across a wider range of inverse problems and degradation types could be further explored.
  • Theoretical Understanding: While the paper provides strong empirical evidence and a metric for spectral bias, deeper theoretical analysis of how these controlled layers precisely influence the learned prior could be beneficial.
  • Future Work: The authors suggest exploring these controlled layers in other network architectures and for more complex inverse imaging problems. They also mention the potential for further investigation into the relationship between spectral bias and network architecture.

Summary updated

Abstract

The deep image prior showed that a randomly initialized network with a suitable architecture can be trained to solve inverse imaging problems by simply optimizing it’s parameters to reconstruct a single degraded image. However, it suffers from two practical limitations. First, it remains unclear how to control the prior beyond the choice of the network architecture. Second, training requires an oracle stopping criterion as during the optimization the performance degrades after reaching an optimum value. To address these challenges we introduce a frequency-band correspondence measure to characterize the spectral bias of the deep image prior, where low-frequency image signals are learned faster and better than high-frequency counterparts. Based on our observations, we propose techniques to prevent the eventual performance degradation and accelerate convergence. We introduce a Lipschitz-controlled convolution layer and a Gaussian-controlled upsampling layer as plug-in replacements for layers used in the deep architectures. The experiments show that with these changes the performance does not degrade during optimization, relieving us from the need for an oracle stopping criterion. We further outline a stopping criterion to avoid superfluous computation. Finally, we show that our approach obtains favorable results compared to current approaches across various denoising, deblocking, inpainting, super-resolution and detail enhancement tasks. Code is available at https://github.com/shizenglin/Measure-and-Control-Spectral-Bias .
BibTeX
@article{shi2022measuring,
  author = {Zenglin Shi and Pascal Mettes and Subhransu Maji and Cees G. M. Snoek},
  title = {{On Measuring and Controlling the Spectral Bias of the Deep Image Prior}},
  journal = {Int. J. Comput. Vis.},
  year = {2022},
  doi = {10.1007/s11263-021-01572-7},
}
Figure