General-Purpose Bayesian Tensor Learning With Automatic Rank Determination and Uncertainty Quantification

Kaiqi Zhang, Cole Hawkins, Zheng Zhang

Introduction

Tensors serve as the fundamental data structure for high-dimensional analysis. However, leveraging tensor factorizations like CP, Tucker, or Tensor-Train involves significant challenges, primarily the NP-hard problem of rank determination and the lack of uncertainty quantification in standard optimization methods. This paper presents a generic Bayesian framework designed to automate rank selection while providing reliable uncertainty estimates across diverse applications, from medical imaging to deep learning.

Methodology

The proposed Generalized Bayesian Tensor Learning framework treats tensor factors as probability distributions rather than fixed point estimates.

Rank-Shrinking Priors

The core of the framework is a hierarchical prior (Gaussian and Gamma distributions) that automatically "shrinks" redundant tensor components toward zero during training. This mechanism enables the model to learn the optimal rank directly from the data, bypassing the traditional manual trial-and-error process.

Scalable Bayesian Inference

To handle large-scale datasets, the authors implement two stochastic solvers:

  1. Stochastic Gradient Hamiltonian Monte Carlo (SGHMC): A physics-inspired sampling method utilizing mini-batches to handle massive data.
  2. Stein Variational Gradient Descent (SVGD): A deterministic approach using a small set of "particles" to approximate the posterior distribution, offering a more memory-efficient alternative to HMC.

The framework is designed to be "plug-and-play," supporting various tensor formats and tasks including completion, regression, and classification.

Experimental Evaluation

The framework was evaluated on synthetic tensors, the PINCAT MRI dataset, Fashion-MNIST (for tensorized fully connected networks), and CIFAR-10 (for tensorized Convolutional Neural Networks).

Rank Recovery and Data Reconstruction

In synthetic benchmarks, the proposed methods successfully identified the ground-truth rank (5), whereas previous state-of-the-art Bayesian methods like BFCP often underestimated the rank, leading to higher error rates. On the MRI dataset, the method achieved significantly lower Root Mean Square Error (RMSE) and more accurate noise estimation than existing benchmarks.

Table 1: Comparison of rank estimation and RMSE across synthetic and MRI datasets, showing the proposed methods' superior rank recovery.

Deep Learning Compression and Performance

The framework demonstrates significant utility in compressing deep neural networks. On the CIFAR-10 dataset, the model achieved a 127× compression ratio for a CNN while maintaining high accuracy (88.41% compared to 91.53% for the uncompressed baseline).

Table 2: Performance comparison on Fashion-MNIST and CIFAR-10 datasets, highlighting compression ratios, log-likelihood (LL), and accuracy improvements over MAP.

Key Insights

  • Beyond Mean-Field Assumptions: By avoiding the restrictive independence assumptions of older methods, this framework succeeds in highly non-linear deep learning tasks where previous Bayesian tensor methods failed.
  • Generalization: The Bayesian approach consistently yielded higher predictive log-likelihood (LL) and accuracy than standard Maximum A Posteriori (MAP) optimization, indicating that accounting for uncertainty helps the model generalize better to unseen data.
  • Efficiency vs. Accuracy: While SVGD is faster and more memory-efficient, it tends to underestimate uncertainty compared to SGHMC.

Conclusion

This framework provides a robust solution for automated tensor learning. While Bayesian training remains more computationally intensive than standard SGD, the benefits in rank determination and uncertainty quantification are substantial for safety-critical fields. Future work will focus on reducing computational overhead and exploring more complex tensor structures.

Summary updated

Abstract

A major challenge in many machine learning tasks is that the model expressive power depends on model size. Low-rank tensor methods are an efficient tool for handling the curse of dimensionality in many large-scale machine learning models. The major challenges in training a tensor learning model include how to process the high-volume data, how to determine the tensor rank automatically, and how to estimate the uncertainty of the results. While existing tensor learning focuses on a specific task, this paper proposes a generic Bayesian framework that can be employed to solve a broad class of tensor learning problems such as tensor completion, tensor regression, and tensorized neural networks. We develop a low-rank tensor prior for automatic rank determination in nonlinear problems. Our method is implemented with both stochastic gradient Hamiltonian Monte Carlo (SGHMC) and Stein Variational Gradient Descent (SVGD). We compare the automatic rank determination and uncertainty quantification of these two solvers. We demonstrate that our proposed method can determine the tensor rank automatically and can quantify the uncertainty of the obtained results. We validate our framework on tensor completion tasks and tensorized neural network training tasks.
BibTeX
@article{zhang2021general,
  author = {Kaiqi Zhang and Cole Hawkins and Zheng Zhang},
  title = {{General-Purpose Bayesian Tensor Learning With Automatic Rank Determination and Uncertainty Quantification}},
  journal = {Frontiers Artif. Intell.},
  year = {2021},
  doi = {10.3389/frai.2021.668353},
}
Figure