Total Deep Variation: A Stable Regularizer for Inverse Problems

Erich Kobler, Alexander Effland, Karl Kunisch, Thomas PockarXiv 2020 · OpenReview

TL;DR

This paper introduces "Total Deep Variation" (TDV), a novel deep learning-based regularizer for inverse problems that combines the strengths of traditional variational methods with the power of neural networks. TDV achieves state-of-the-art results across various imaging tasks by offering a mathematically rigorous and stable framework for learning image priors.

What Problem Does This Paper Address?

Many real-world problems, such as image denoising, super-resolution, and medical imaging reconstruction, can be framed as "inverse problems." These problems involve recovering an original image from degraded or incomplete measurements. A common approach is to use variational methods, which balance fitting the observed data with enforcing prior knowledge about the image's structure (regularization). While handcrafted regularizers (like Total Variation) have been successful, they often struggle to capture the complexity of natural images. Deep learning methods have shown great promise in outperforming these handcrafted approaches, but they often lack the theoretical guarantees and interpretability of variational methods. This paper aims to bridge this gap by developing a data-driven regularizer that is both powerful and theoretically grounded.

Methods

The core of the proposed method is the Total Deep Variation (TDV) regularizer. TDV is implemented as a convolutional neural network (CNN) designed to extract features at multiple scales and across successive blocks, mimicking the structure of modern deep networks. This CNN acts as a learned prior in the variational framework.

The training of the TDV regularizer is formulated as a mean-field optimal control problem. This involves minimizing a cost functional that represents the expected loss over a distribution of training data. The authors derive a gradient flow equation for the image reconstruction process and then optimize the stopping time of this flow and the parameters of the TDV regularizer. This approach allows for a rigorous mathematical analysis, including proving the existence of solutions and deriving stability guarantees.

Key aspects of their method include:

  • Residual Multi-Scale Network: The TDV regularizer is built upon a residual multi-scale network architecture, which is known for its effectiveness in capturing hierarchical image features.
  • Mean-Field Optimal Control: This framework allows for learning the regularizer parameters by considering the expected behavior over a distribution of data, leading to more robust and generalizable solutions.
  • Stability Analysis: The authors provide theoretical guarantees for the stability of their method with respect to input perturbations (noise) and parameter variations, which is crucial for reliable inverse problem solvers.

Datasets

The paper uses the BSDS400 dataset for training the TDV regularizer across various experiments. For evaluation, they also mention using:

  • BSDS68 dataset
  • DIV2K validation set
  • MAYO dataset (for CT reconstruction)

No new benchmarks are introduced; instead, the authors evaluate their method on standard datasets for the respective tasks.

Key Insights and Ideas

  • Bridging Variational and Deep Learning: The TDV regularizer successfully merges the interpretability and theoretical guarantees of variational methods with the representational power of deep neural networks. This is a significant step towards more robust and trustworthy deep learning solutions for inverse problems.
  • Learned Priors Adapt to Tasks: The eigenfunctions analysis reveals that the learned TDV regularizers are "discriminative priors," meaning they adapt their behavior to the specific imaging task (e.g., denoising vs. super-resolution). This is a key advantage over generic handcrafted priors.
  • Optimal Stopping Time is Crucial: The paper highlights the importance of finding an optimal stopping time for the gradient flow, which balances reconstruction quality and prevents oversmoothing or artifact generation.
  • Robustness is Achievable: The theoretical and numerical stability analyses demonstrate that the proposed TDV method is robust to input perturbations and variations in learned parameters, as well as adversarial attacks.

Key Results

The paper reports state-of-the-art results on several imaging tasks, including:

  • Additive Gaussian Denoising: Achieved competitive or superior PSNR values compared to existing methods, often with significantly fewer trainable parameters. For example, their TDV3,25 model achieved high PSNR values on various datasets, outperforming many complex networks.
  • Computed Tomography (CT) Reconstruction: Demonstrated impressive results for undersampled CT reconstruction, effectively suppressing artifacts and preserving fine details without task-specific retraining.
  • Magnetic Resonance Imaging (MRI) Reconstruction: Showed versatility by applying the TDV regularizer to undersampled MRI data, removing artifacts and achieving good reconstructions.
  • Single Image Super-Resolution (SISR): Achieved comparable or better PSNR values than state-of-the-art networks with similar parameter counts, showcasing the learned regularizer's ability to recover high-frequency details.

Limitations and Future Work

  • Computational Cost: While the TDV regularizer is efficient, the training process involving optimal control can be computationally intensive.
  • Generalization Bounds: The paper derives upper bounds for generalization error, but these are noted to be potentially loose, especially for certain types of images.
  • Future Work: The authors suggest exploring more complex network architectures for the TDV regularizer, investigating its application to other inverse problems, and further refining the theoretical analysis of generalization bounds. They also highlight the potential for TDV to be applied in scenarios with limited training data.

Summary updated

Abstract

Various problems in computer vision and medical imaging can be cast as inverse problems. A frequent method for solving inverse problems is the variational approach, which amounts to minimizing an energy composed of a data fidelity term and a regularizer. Classically, handcrafted regularizers are used, which are commonly outperformed by state-of-the-art deep learning approaches. In this work, we combine the variational formulation of inverse problems with deep learning by introducing the data-driven general-purpose total deep variation regularizer. In its core, a convolutional neural network extracts local features on multiple scales and in successive blocks. This combination allows for a rigorous mathematical analysis including an optimal control formulation of the training problem in a mean-field setting and a stability analysis with respect to the initial values and the parameters of the regularizer. In addition, we experimentally verify the robustness against adversarial attacks and numerically derive upper bounds for the generalization error. Finally, we achieve state-of-the-art results for numerous imaging tasks.
BibTeX
@misc{kobler2020total,
  author = {Erich Kobler and Alexander Effland and Karl Kunisch and Thomas Pock},
  title = {{Total Deep Variation: A Stable Regularizer for Inverse Problems}},
  year = {2020},
  eprint = {2006.08789},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.08789},
}
Figure