Robust Tensor Factorization for Color Image and Grayscale Video Recovery

Shiqiang Du, Yuqing Shi, Wenjin Hu, Weilan Wang, Jing LianIEEE Access 2020 · OpenReview

Abstract

Low-rank tensor completion (LRTC) plays an important role in many fields, such as machine learning, computer vision, image processing, and mathematical theory. Since rank minimization is an NP-hard problem, one strategy is that it is converted into a convex relaxation tensor nuclear norm (TNN) that requires the repeated calculation of time-consuming SVD, and the other is to convert it into some product of two smaller tensors that are easy to fall into the local minimum. In order to overcome the above shortcomings, we propose a robust tensor factorization (RTF) model for solving LRTC. In RTF, the noisy tensor data with missing entries is decomposed into low-rank tensor and noisy tensor, and then the low-rank tensor is equivalently decomposed into t-products (essentially vectors convolution) of two smaller tensors: orthogonal dictionary tensor and low-rank representation tensor. Meanwhile, the TNN of low-rank representation tensor is adopted to characterize the low-rank structure of the tensor data for preserving global information. Then, an effective iterative update algorithm based on the alternating direction method of multipliers (ADMM) is proposed to solve RTF. Finally, numerical experiments on image recovering and video completion tasks show the effectiveness of the proposed RTF model compared with several state-of-the-art tensor completion models.

Introduction

Multidimensional data, such as color images and video sequences, often suffer from missing entries due to sensor failures, transmission errors, or environmental noise. Low-Rank Tensor Completion (LRTC) has emerged as a powerful framework for recovering this missing data by exploiting inherent correlations. However, existing LRTC methods typically face a significant trade-off between accuracy and computational efficiency.

Nuclear norm-based methods capture global structural information effectively but are computationally expensive due to repeated Singular Value Decompositions (SVD) on large tensors. Conversely, factorization-based methods are faster as they decompose large tensors into smaller factors, but they often converge to local minima and fail to preserve the global structural integrity of the original data. This paper introduces Robust Tensor Factorization (RTF), a framework designed to bridge this gap by combining the global benefits of the Tensor Nuclear Norm (TNN) with the efficiency of tensor factorization.

Methodology

The core of the proposed RTF model is a hybrid approach to tensor decomposition. The process involves several key steps:

  1. Decomposition: The observed noisy and incomplete tensor is first split into a low-rank component and a sparse noise component (modeled using the $\ell_1$-norm to handle "salt-and-pepper" noise).
  2. t-product Factorization: The low-rank component is further decomposed using a specialized "t-product" (a form of tensor convolution) into an orthogonal dictionary tensor and a low-rank representation tensor.
  3. Hybrid Constraint: Instead of applying the TNN to the massive original tensor, it is applied to the smaller representation tensor. This preserves global information while significantly reducing the size of the matrices involved in SVD calculations.
  4. Optimization: The model is solved using the Alternating Direction Method of Multipliers (ADMM), ensuring stable convergence.
Illustration of the tensor factorization framework using the t-product.

By forcing the dictionary tensor to be orthogonal, the algorithm achieves mathematical simplifications that enhance robustness and computational speed. This "best of both worlds" approach provides the mathematical rigor of the nuclear norm without the prohibitive costs associated with full-scale data processing.

Experimental Results

The RTF model was evaluated against state-of-the-art methods using two standard benchmarks: the Kodak Dataset (24 high-quality color images) and YUV Video Sequences (18 grayscale sequences).

Performance and Accuracy

In noiseless image recovery tests, RTF consistently achieved the highest Peak Signal-to-Noise Ratio (PSNR). It outperformed the leading alternative, TRPCA, by approximately 0.40 to 0.68 dB across various sampling rates (ranging from 10% to 50% known pixels). Qualitatively, RTF demonstrated superior visual fidelity, successfully recovering complex textures and fine details (such as eyes or foliage) even when 80-90% of the data was missing.

Computational Efficiency

One of the most significant advantages of RTF is its speed. In image restoration tasks, RTF was nearly 4 times faster than TRPCA. This efficiency stems directly from performing SVD operations on the smaller factor tensors rather than the original high-dimensional data. For video sequences, RTF also maintained better temporal smoothness and fewer artifacts compared to factorization-only methods.

Conclusion and Future Work

The Robust Tensor Factorization model provides a scalable and accurate solution for multidimensional data recovery. While highly effective, the authors noted challenges in scaling to very large video datasets (beyond 30 frames) due to memory constraints. Future research directions include integrating this tensor-based approach with deep learning frameworks to further enhance recovery performance in highly complex or dynamic scenes and refining parameter tuning for diverse noise environments.

Summary updated

Abstract

Low-rank tensor completion (LRTC) plays an important role in many fields, such as machine learning, computer vision, image processing, and mathematical theory. Since rank minimization is an NP-hard problem, one strategy is that it is converted into a convex relaxation tensor nuclear norm (TNN) that requires the repeated calculation of time-consuming SVD, and the other is to convert it into some product of two smaller tensors that are easy to fall into the local minimum. In order to overcome the above shortcomings, we propose a robust tensor factorization (RTF) model for solving LRTC. In RTF, the noisy tensor data with missing entries is decomposed into low-rank tensor and noisy tensor, and then the low-rank tensor is equivalently decomposed into t-products (essentially vectors convolution) of two smaller tensors: orthogonal dictionary tensor and low-rank representation tensor. Meanwhile, the TNN of low-rank representation tensor is adopted to characterize the low-rank structure of the tensor data for preserving global information. Then, an effective iterative update algorithm based on the alternating direction method of multipliers (ADMM) is proposed to solve RTF. Finally, numerical experiments on image recovering and video completion tasks show the effectiveness of the proposed RTF model compared with several state-of-the-art tensor completion models.
BibTeX
@inproceedings{du2020robust,
  author = {Shiqiang Du and Yuqing Shi and Wenjin Hu and Weilan Wang and Jing Lian},
  title = {{Robust Tensor Factorization for Color Image and Grayscale Video Recovery}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=ePvv4Hpe5U},
}
Figure