Principal Component Wavelet Networks for Solving Linear Inverse Problems

Bernard Tiddeman, Morteza GhahremaniSymmetry 2021 · DOI · OpenReview

TL;DR

This paper introduces Principal Component Wavelet Networks (PCWNs), a novel deep learning architecture that combines wavelet transforms with Principal Component Analysis (PCA) to efficiently solve linear inverse problems. PCWNs offer a fast, deterministic, and robust alternative to existing methods like GANs, achieving competitive or superior performance on tasks such as image denoising, inpainting, and super-resolution.

What Problem Does This Paper Address?

Linear inverse problems are ubiquitous in fields like image processing. These problems are characterized by a forward process that is easy to compute but an inverse process that is difficult and often ill-posed. A common example is reconstructing an original signal $x$ from noisy measurements $y$, where $y = Ax + n$. Traditional approaches often rely on hand-crafted priors (like sparsity in a wavelet domain) to regularize the problem, but these priors can be too generic, leading to artifacts. More recent deep learning methods, particularly Generative Adversarial Networks (GANs), have shown impressive results but are notoriously difficult to train, prone to overfitting, and sensitive to training perturbations. This paper aims to bridge the gap between traditional wavelet methods and complex deep learning approaches by proposing a simpler, more robust, and efficiently trainable network for solving these inverse problems.

Methods

The core innovation is the Principal Component Wavelet Network (PCWN). At each level of decomposition, PCWNs apply a predetermined wavelet filter bank (specifically, separable derivative of Gaussian filters) to the input channels. Crucially, instead of directly using all the resulting subbands, Principal Component Analysis (PCA) is applied across these subbands to learn a set of 1x1 convolutional filters. These PCA-derived filters act as a dimensionality reduction mechanism, controlling the growth of channels and effectively learning a compressed representation.

The authors demonstrate that this architecture is mathematically equivalent to a separable Convolutional Neural Network (CNN). The mean subtraction in PCA corresponds to bias terms, and the principal components form the convolutional filters. This equivalence allows PCWNs to leverage the power of CNNs while using a deterministic and computationally efficient training process based on PCA, rather than backpropagation.

For reconstruction, PCWNs employ an autoencoder-like structure, performing the decomposition steps in reverse. The training algorithm is deterministic and involves calculating means and covariance matrices from the training data to derive the PCA filters. This approach avoids the complexities and potential issues associated with adversarial training.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Face Images: The Labelled Faces in the Wild (LFW) dataset was used for training and testing face-related inverse problems. Images were cropped and downsampled to 64x64.
  • ImageNet: For general image processing tasks like inpainting and super-resolution, the ImageNet 2012 dataset was used. The model was trained on the full 1.2M training set and tested on a separate set of 3000 validation images, downsampled to 64x64.

The paper does not introduce new benchmarks but compares its performance against existing methods on these standard datasets.

Key Insights and Ideas

  • PCA as a CNN Filter Learner: The paper elegantly shows how PCA, when applied to wavelet subbands in a specific way, naturally leads to an architecture equivalent to a CNN, with PCA learning the convolutional filters. This provides a principled way to learn filters without relying on backpropagation.
  • Deterministic and Robust Training: By using PCA, the PCWN training is deterministic and robust to overfitting, a significant advantage over GAN-based methods which are known for their training instability.
  • Bridging Wavelets and Deep Learning: PCWNs effectively combine the strengths of traditional wavelet transforms (e.g., multi-resolution analysis) with the learning capabilities of deep neural networks, creating a powerful hybrid approach.
  • Efficient Autoencoder Construction: The inherent structure of PCWNs allows for a simple and direct construction of deep autoencoders, facilitating reconstruction tasks.

Key Results

The PCWN method demonstrated strong performance across various linear inverse problems:

  • Face Image Tasks: On face images, the PCWN system achieved competitive results, outperforming the original OneNet on Blockwise Inpainting (BI) and Pixelwise Inpaint Denoising (PID) tasks, and showing comparable performance to DU-ADMM. For compressive sensing (CS) on face images, it achieved a PSNR of 29.60 ± 2.3.
  • ImageNet Tasks: For ImageNet, the PCWN system showed improved or equivalent performance to existing methods on inpainting and super-resolution tasks. For example, on blockwise inpainting, it achieved a PSNR of 26.89 ± 2.3, and on super-resolution, 27.92 ± 2.5.
  • Training Speed: A significant advantage highlighted is the training speed. While methods like OneNet and DU-ADMM require tens of thousands or even hundreds of thousands of iterations and can take days or weeks to train, PCWNs can be trained in a matter of hours (e.g., 4 hours for the celeb-a dataset, 15 hours for the ImageNet training partition).

Limitations and Future Work

  • Parameter Selection: The authors acknowledge that selecting optimal parameters (e.g., the percentage of variance to retain in PCA) can be challenging and problem-dependent. Future work will focus on learning these parameters from data.
  • Architecture Exploration: The current PCWN architecture is pyramidal. Future research will explore more complex architectures, including non-linear activation functions, batch normalization, and other CNN components, while ensuring invertibility for reconstruction.
  • Broader Applications: The PCWN framework is seen as a general-purpose solution and has potential for applications beyond linear inverse problems, such as colorization, classification, and outlier detection.
  • Mathematical Analysis: The deterministic nature of PCWNs makes them more amenable to mathematical analysis compared to stochastic methods, which is an area for future exploration.

Summary updated

Abstract

In this paper we propose a novel learning-based wavelet transform and demonstrate its utility as a representation in solving a number of linear inverse problems—these are asymmetric problems, where the forward problem is easy to solve, but the inverse is difficult and often ill-posed. The wavelet decomposition is comprised of the application of an invertible 2D wavelet filter-bank comprising symmetric and anti-symmetric filters, in combination with a set of 1×1 convolution filters learnt from Principal Component Analysis (PCA). The 1×1 filters are needed to control the size of the decomposition. We show that the application of PCA across wavelet subbands in this way produces an architecture equivalent to a separable Convolutional Neural Network (CNN), with the principal components forming the 1×1 filters and the subtraction of the mean forming the bias terms. The use of an invertible filter bank and (approximately) invertible PCA allows us to create a deep autoencoder very simply, and avoids issues of overfitting. We investigate the construction and learning of such networks, and their application to linear inverse problems via the Alternating Direction of Multipliers Method (ADMM). We use our network as a drop-in replacement for traditional discrete wavelet transform, using wavelet shrinkage as the projection operator. The results show good potential on a number of inverse problems such as compressive sensing, in-painting, denoising and super-resolution, and significantly close the performance gap with Generative Adversarial Network (GAN)-based methods.
BibTeX
@article{tiddeman2021principal,
  author = {Bernard Tiddeman and Morteza Ghahremani},
  title = {{Principal Component Wavelet Networks for Solving Linear Inverse Problems}},
  journal = {Symmetry},
  year = {2021},
  doi = {10.3390/sym13061083},
}
Figure