Degradation Accordant Plug-and-Play for Low-Rank Tensor Completion

Yexun Hu, Tai-Xiang Jiang, Xi-Le ZhaoIJCAI 2022 · DOI

TL;DR

This paper introduces a novel method for reconstructing incomplete multidimensional data (like color images or videos) by combining a traditional low-rank tensor prior with a data-driven prior learned by a deep neural network. This "Degradation Accordant Plug-and-Play" (DAP-LRTC) approach leverages the strengths of both methods to achieve superior reconstruction quality and generalization across different data types.

What Problem Does This Paper Address?

Multidimensional data, such as color images, videos, and medical scans, often have missing information due to acquisition limitations or storage constraints. Tensor completion aims to fill these gaps. While traditional methods exploit the inherent low-rank structure of such data, they often struggle to preserve fine spatial details. Conversely, deep learning models can capture complex image priors but might not fully leverage the global low-rank structure. This paper addresses the challenge of effectively combining these two complementary approaches to achieve more accurate and detailed tensor completion.

Methods

The core of this work is the proposed DAP-LRTC model, which integrates two key priors:

  • Low-Rank Prior: The Tensor Nuclear Norm (TNN) is used to capture the overall low-dimensionality of the tensor data. This is a standard technique in tensor completion, ensuring that the reconstructed tensor maintains a globally coherent structure.
  • Data-Driven Prior (Implicit Regularizer): This is the novel aspect. Instead of a fixed mathematical regularizer, the authors formulate an implicit regularizer that is solved using a deep convolutional neural network (CNN) within a Plug-and-Play (PnP) framework. This CNN is pretrained for image inpainting on a large dataset of natural images. The idea is that this pretrained inpainter can effectively express a "data-driven prior" that aligns with the degradation process of the observed data, thereby preserving fine details and textures.

The optimization problem is solved using an Alternating Direction Method of Multipliers (ADMM) algorithm. The ADMM framework allows for the decoupling of the TNN subproblem (solved analytically using t-SVD) and the implicit regularizer subproblem, which is delegated to the pretrained CNN inpainter. This PnP approach allows for the flexible integration of off-the-shelf deep learning models.

Datasets

The paper evaluates its method on various types of multidimensional imaging data:

  • Color Images: Standard color images are used for testing structural missing patterns.
  • Videos: Several videos are used to evaluate performance under random and structural missing data.
  • Multispectral Images (MSIs): MSI datasets are used to assess performance with different sampling rates.
  • MRI Data: MRI data is used for parameter analysis and convergence studies.

The paper does not introduce new datasets but utilizes publicly available ones for evaluation.

Key Insights and Ideas

  • Synergy of TNN and Deep Inpainting: The paper demonstrates that combining the global structural prior of TNN with the local detail-preserving prior of a deep inpainting CNN leads to superior results compared to methods relying on only one of these priors.
  • Degradation Accordance: By using a CNN pretrained for inpainting, the method implicitly aligns with the degradation process of the observed data, making it more effective than using a generic denoising CNN.
  • Plug-and-Play Flexibility: The ADMM framework allows for easy integration of different CNN architectures as the implicit regularizer, offering flexibility for various image restoration tasks beyond just completion.
  • Generalization Ability: The method shows remarkable generalization capabilities, performing well across different types of multidimensional data (images, videos, MSIs) even though the CNN inpainter is trained on natural images.

Key Results

  • Quantitative Improvements: The proposed DAP-LRTC method consistently outperforms state-of-the-art tensor completion methods (e.g., HaLRTC, TNN, DCTNN, DP3LRTC, OPN) in terms of PSNR and SSIM metrics across color images, videos, and MSIs. For instance, on color images with structural missing, DAP-LRTC achieves PSNRs up to 51.89 and SSIMs up to 0.995, outperforming others.
  • Qualitative Superiority: Visual results show that DAP-LRTC reconstructs finer details and textures, especially in enlarged areas, and better preserves structural integrity (e.g., in video frames) compared to other methods.
  • Efficiency: While the deep learning component adds computational cost, the ADMM framework allows for efficient updates. The running times are reported, and while not always the fastest, the performance gains justify the computational trade-off.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that the method is "still time-consuming," largely due to the deep learning component. This is identified as a direction for future work.
  • CNN Training: While the paper leverages pretrained CNNs, the convergence of the ADMM algorithm is guaranteed if the CNN is "properly trained." Further research could explore optimal CNN training strategies for this specific application.
  • Extension to Other Tasks: The authors suggest that the framework is not limited to tensor completion and can be extended to other multidimensional image restoration tasks like video super-resolution or deblurring by adapting the degradation operator and the CNN.

Summary updated

BibTeX
@inproceedings{hu2022degradation,
  author = {Yexun Hu and Tai-Xiang Jiang and Xi-Le Zhao},
  title = {{Degradation Accordant Plug-and-Play for Low-Rank Tensor Completion}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/256},
}
Figure