Bayesian-Driven Learning of A New Weighted Tensor Norm for Tensor Recovery

Jingjing Zheng, Yankai CaoICLR 2024 · OpenReview

TL;DR

Researchers have developed a new bilevel framework for tensor completion that automatically learns a data-dependent weighted tensor norm using Bayesian Optimization (BO). By treating the selection of weight parameters as an upper-level "black-box" problem, the method significantly outperforms existing state-of-the-art techniques, especially when dealing with data that has imbalanced low-rank structures or abrupt, non-smooth changes.

What Problem Does This Paper Address?

Tensors—multi-dimensional arrays used to represent data like color images (height x width x color) or videos (height x width x color x time)—are often incomplete due to sensor failure or limited sampling. "Tensor recovery" aims to fill in these missing values by assuming the underlying data is "low-rank" (meaning it has a simpler, redundant structure).

Current state-of-the-art methods often rely on the tensor Singular Value Decomposition (t-SVD). However, these methods face two major hurdles:

  1. Imbalanced Low-Rankness: Real-world data doesn't always have the same level of redundancy across all its dimensions.
  2. Non-Smooth Changes: Most methods use fixed mathematical transforms (like the Discrete Fourier Transform) that assume data changes smoothly. If a video has a sudden scene cut or an image has sharp edges, these fixed transforms struggle to recover the data accurately.
Examples of challenging tensor data structures.

Methods

The authors introduce a bilevel optimization model to make tensor recovery more flexible and data-aware.

  • The New Norm: They propose a novel weighted tensor norm that considers different "unfoldings" and projections of the tensor. Unlike previous models where weights were fixed or equal, this model allows weights to be tuned to the specific characteristics of the data.
  • Bilevel Learning:
    • Lower Level: Given a set of weights, the model solves a standard tensor completion problem using an algorithm called the APMM (Alternating Proximal Method of Multipliers).
  • Upper Level: This is the "brain" of the operation. It treats the weight selection as a black-box optimization problem. Since evaluating every possible weight combination is too expensive, they use Bayesian Optimization (BO) with a "knowledge gradient" acquisition function to efficiently find the best weights.
  • Efficiency Trick: To speed up the learning process, the authors sample small "slices" of the observed data to train the weights before applying them to the full-scale tensor recovery.

Datasets

The method was evaluated using the Berkeley Segmentation Dataset (BSD).

  • The researchers randomly selected 50 color images from the dataset.
  • They tested the model across three different sampling rates (0.3, 0.5, and 0.7), representing scenarios where 70%, 50%, and 30% of the data is missing, respectively.

Key Insights and Ideas

  • Moving Beyond Fixed Transforms: By learning the weights and transforms from the data itself rather than relying on a fixed Fourier transform, the model adapts to "jumpy" or non-continuous data that typically breaks other t-SVD methods.
  • BO as a Hyperparameter Tuner: Tuning weights for high-order tensors is notoriously difficult as the number of dimensions increases. Using BO transforms a manual, "guess-and-check" process into an automated, mathematically principled search.
  • Convexity Matters: The authors proved that their proposed norm remains convex (provided weights are non-negative), which ensures that the lower-level optimization problem can be solved reliably and reach a global optimum.

Key Results

The proposed method, labeled "Ours-BO," consistently outperformed five other state-of-the-art methods (including TNN-DCT, SNN, and WSTNN) across all tested sampling rates.

  • Quantitative Gains: At a 30% sampling rate (0.3), "Ours-BO" achieved a Peak Signal-to-Noise Ratio (PSNR) of 28.33 dB, compared to 25.75 dB for the next best existing method (WSTNN)—a significant jump in image reconstruction quality.
  • The "BO" Advantage: Comparing their method with equal weights ("Ours") versus BO-learned weights ("Ours-BO") showed an average improvement of about 0.5 dB, proving that the Bayesian learning step adds tangible value over simple averaging.
  • Overall Superiority: Across all tests, the proposed framework demonstrated a superiority of at least 2 dB over the broader field of existing competitors.

Limitations and Future Work

As this was published as a "Tiny Paper," some details are condensed:

  • Computational Complexity: While BO is more efficient than a grid search, the paper does not extensively discuss the total wall-clock time required for the bilevel optimization compared to simpler, single-level methods.
  • Scalability: The current tests were performed on 2D images (treated as 3D tensors). It remains to be seen how well the BO search scales when dealing with much higher-order tensors (e.g., 5D or 6D scientific data) where the number of weight parameters would grow.
  • Broader Applications: The authors suggest that this framework isn't limited to tensor completion; it could potentially be adapted for other recovery tasks like tensor de-noising or principal component analysis.

Summary updated

Abstract

This study addresses the performance limitations of t-SVD-based tensor recovery caused by non-smooth changes and imbalanced low-rankness in tensor data. We introduce a novel bilevel tensor completion model, integrating the learning of a data-dependent weighted tensor norm within the tensor completion framework as an upper-level problem. We treat the optimization of the bilevel problem as a black-box problem, employing Bayesian Optimization (BO) for efficient learning of the proposed tensor norm. Numerical experiments demonstrated the superior performance of our proposed method compared to state-of-the-art methods in tensor completion. The code of our method is available at \url{https://github.com/jzheng20/TR-BO.git}.
BibTeX
@inproceedings{zheng2024bayesian,
  author = {Jingjing Zheng and Yankai Cao},
  title = {{Bayesian-Driven Learning of A New Weighted Tensor Norm for Tensor Recovery}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=ciEbMa2xuC},
}
Figure