Discriminative Transfer Learning for General Image Restoration

Lei Xiao, Felix Heide, Wolfgang Heidrich, Bernhard Schölkopf, Michael Hirsch

Abstract

Discriminative learning approaches have achieved a convincing trade-off between image quality and computational efficiency in restoration tasks. However, these methods typically require separate training for each specific task and condition, leading to a "brittle" specialization. We present Discriminative Transfer Learning (DTL), a framework that incorporates formal proximal optimization with discriminative learning. DTL allows for a single-pass training that can be reused across various restoration problems—such as denoising, deblurring, and inpainting—while maintaining the speed of specialized discriminative models.

Introduction: Bridging the Gap

In computer vision, image restoration has long been divided between two philosophies. Generative/classical models are flexible and modular but computationally expensive. Conversely, discriminative models (like early neural networks) are fast but specialized; they often require a unique model for every task or noise level. DTL offers a middle ground: the speed of discriminative learning with the "plug-and-play" modularity of classical optimization.

The Problem: Specialized vs. Versatile Models

Traditional discriminative models are highly specialized. A model trained for denoising at a specific noise level ($\sigma=15$) often fails when applied to $\sigma=25$ or a different task like deblurring. This necessitates storing dozens of models for different scenarios, which is impractical for real-world applications like mobile photography. The challenge lies in building a model as fast as a specialized network but as versatile as a general-purpose optimization algorithm.

Methodology: Decoupling Physics from Statistics

The core innovation of DTL is the decoupling of the image restoration objective into two alternating subproblems using Half-Quadratic Splitting (HQS):

  1. The Data Proximal Operator: This handles the task-specific physics (e.g., a specific blur kernel). It is mathematically well-defined and does not require learning.
  2. The Prior Proximal Operator: This handles the "naturalness" of the image. The authors realized that while the data operator changes per task, the prior operator can act as a universal Gaussian denoiser.
The DTL framework decouples task-specific data operators from a shared, learned prior proximal operator across iterations.

Instead of a slow handcrafted prior, DTL uses a modified Trainable Nonlinear Reaction Diffusion (TRD) model as the prior operator. By using a recurrent architecture, the same learned parameters are reused across multiple iterations, keeping the model compact and ensuring convergence.

Key Results and Performance

DTL demonstrates significant robustness compared to specialized models. While specialized TRD models see a sharp decline in performance when test noise deviates from training noise, DTL maintains stable and superior PSNR across a range of conditions.

Comparison of PSNR performance across different noise levels. DTL shows superior robustness compared to specialized TRD models (TRD15, TRD25).
  • Versatility: A single DTL model can be transferred to solve untrained tasks like inpainting or deblurring by simply swapping the data operator at test time.
  • Competitive Quality: In denoising, DTL achieves PSNR results (~31.00 dB) comparable to state-of-the-art specialized methods like BM3D. In non-blind deconvolution, it achieved an average PSNR of 34.34 dB on the Levin benchmark, outperforming previous methods like CSF (33.48 dB).
  • Efficiency: Implemented in Halide, DTL processes a $512 \times 512$ image in just 0.26 seconds, compared to 111 seconds for optimization-based methods like EPLL.

Conclusion

Discriminative Transfer Learning successfully bridges the gap between modularity and speed. By embedding a discriminative prior within a formal optimization loop, DTL provides a "universal" restorer that is both fast and flexible. Future work may extend this approach to deeper architectures, potentially leading to even more powerful and compact universal image priors.

Summary updated

Abstract

Recently, several discriminative learning approaches have been proposed for effective image restoration, achieving convincing trade-off between image quality and computational efficiency. However, these methods require separate training for each restoration task (e.g., denoising, deblurring, demosaicing) and problem condition (e.g., noise level of input images). This makes it time-consuming and difficult to encompass all tasks and conditions during training. In this paper, we propose a discriminative transfer learning method that incorporates formal proximal optimization and discriminative learning for general image restoration. The method requires a single-pass training and allows for reuse across various problems and conditions while achieving an efficiency comparable to previous discriminative approaches. Furthermore, after being trained, our model can be easily transferred to new likelihood terms to solve untrained tasks, or be combined with existing priors to further improve image restoration quality.
BibTeX
@article{xiao2018discriminative,
  author = {Lei Xiao and Felix Heide and Wolfgang Heidrich and Bernhard Schölkopf and Michael Hirsch},
  title = {{Discriminative Transfer Learning for General Image Restoration}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=oUOs7CFp0w8},
}
Figure