Learning Deep Analysis Dictionaries for Image Super-Resolution

Jun-Jie Huang, Pier Luigi DragottiIEEE TSP 2020 · OpenReview

Abstract

Inspired by the recent success of deep neural networks and efforts to develop multi-layer dictionary models, we propose a Deep Analysis dictionary Model (DeepAM) optimized for single image super-resolution. Unlike other multi-layer models, our architecture contains $L$ layers of analysis dictionaries and soft-thresholding operators to extract high-level features, followed by a synthesis dictionary layer for regression. Each analysis dictionary is partitioned into an Information Preserving Analysis Dictionary (IPAD) and a Clustering Analysis Dictionary (CAD). The IPAD passes key information to the next layer, while the CAD produces sparse feature representations. DeepAM utilizes both supervised and unsupervised learning. Results show DeepAM outperforms comparable DNNs on small training datasets and adapts to unseen noise levels via simple threshold rescaling.

Introduction

Single Image Super-Resolution (SISR) is a fundamental task in computer vision. While Deep Neural Networks (DNNs) have achieved state-of-the-art results, they often function as "black boxes" requiring vast amounts of training data. Moreover, DNNs are typically sensitive to noise levels; a model trained on clean data often fails when presented with noisy test images, necessitating costly retraining. This paper introduces DeepAM, an interpretable, data-efficient, and noise-robust alternative that leverages the principles of sparse representation within a deep architecture.

The DeepAM Architecture

DeepAM is structured as an $L$-layer deep analysis dictionary model. Each of the first $L$ layers performs a forward pass consisting of an analysis dictionary multiplication followed by a soft-thresholding operator. This sequence gradually extracts high-level features from the input low-resolution (LR) image. The final layer is a synthesis dictionary ($D$) designed to reconstruct the high-resolution (HR) output ($\hat{y}$).

The DeepAM architecture consisting of $L$ analysis layers and a final synthesis layer.

Unlike synthesis-based dictionary models that require iterative sparse pursuit during inference, DeepAM's analysis-based approach allows for efficient computation via simple matrix multiplications and thresholding. This makes the model significantly faster during both training and deployment.

Dictionary Partitioning: IPAD and CAD

A key innovation of DeepAM is the partitioning of each analysis dictionary $\Omega$ into two functional components: the Information Preserving Analysis Dictionary (IPAD) and the Clustering Analysis Dictionary (CAD).

Structure of a single analysis layer partitioned into IPAD ($\Omega_I$) and CAD ($\Omega_C$).
  1. IPAD ($\Omega_I$): Learned using unsupervised methods, this component ensures that the essential signal subspace of the input is preserved and passed to the subsequent layer.
  2. CAD ($\Omega_C$): Learned through supervised methods, this component focuses on identifying discriminative high-frequency features and residuals that are crucial for reconstruction but often lost during downsampling.

This hybrid learning strategy allows the model to maintain a clear functional role for different "neurons," providing a level of interpretability often missing in standard deep learning models.

Experimental Results

The model was evaluated on standard benchmarks such as Set5 and Set14. DeepAM demonstrates a significant advantage in small-data regimes. When trained on only 5,000 patches, DeepAM achieved a PSNR of 35.96 dB on Set5, outperforming comparable DNN architectures with ReLU (35.06 dB) or soft-thresholding (35.62 dB).

Furthermore, DeepAM exhibits unique robustness to noise. Because the soft-thresholding parameters are tied to signal and noise variance, a pre-trained DeepAM can be adapted to new noise levels simply by mathematically rescaling the thresholds of the first layer. In tests with varying noise levels, this rescaled DeepAM maintained high performance, whereas standard DNNs saw their PSNR drop significantly due to the mismatch between training and testing noise.

Conclusion

DeepAM bridges the gap between the interpretability of sparse coding and the representational power of deep learning. Its ability to perform well with limited data and its inherent flexibility in handling noise make it a compelling alternative to standard black-box DNNs for image restoration. Future research will explore extending this framework to fully convolutional architectures and applying it to other domains like medical imaging (CT/MRI), where data efficiency and noise robustness are critical.

Summary updated

Abstract

Inspired by the recent success of deep neural networks and the recent efforts to develop multi-layer dictionary models, we propose a Deep Analysis dictionary Model (DeepAM) which is optimized to address a specific regression task known as single image super-resolution. Contrary to other multi-layer dictionary models, our architecture contains L layers of analysis dictionary and soft-thresholding operators to gradually extract high-level features and a layer of synthesis dictionary which is designed to optimize the regression task at hand. In our approach, each analysis dictionary is partitioned into two sub-dictionaries: an Information Preserving Analysis Dictionary (IPAD) and a Clustering Analysis Dictionary (CAD). The IPAD together with the corresponding soft-thresholds is designed to pass the key information from the previous layer to the next layer, while the CAD together with the corresponding soft-thresholding operator is designed to produce a sparse feature representation of its input data that facilitates discrimination of key features. DeepAM uses both supervised and unsupervised setup. Simulation results show that the proposed deep analysis dictionary model achieves better performance compared to a deep neural network that has the same structure and is optimized using back-propagation when training datasets are small. On noisy image super-resolution, DeepAM can be well adapted to unseen testing noise levels by rescaling the IPAD and CAD thresholds of the first layer.
BibTeX
@article{huang2020learning,
  author = {Jun-Jie Huang and Pier Luigi Dragotti},
  title = {{Learning Deep Analysis Dictionaries for Image Super-Resolution}},
  journal = {IEEE Trans. Signal Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=tqeTR0gtuyCX},
}
Figure