Introduction
Tensors serve as the fundamental data structure for high-dimensional analysis. However, leveraging tensor factorizations like CP, Tucker, or Tensor-Train involves significant challenges, primarily the NP-hard problem of rank determination and the lack of uncertainty quantification in standard optimization methods. This paper presents a generic Bayesian framework designed to automate rank selection while providing reliable uncertainty estimates across diverse applications, from medical imaging to deep learning.
Methodology
The proposed Generalized Bayesian Tensor Learning framework treats tensor factors as probability distributions rather than fixed point estimates.
Rank-Shrinking Priors
The core of the framework is a hierarchical prior (Gaussian and Gamma distributions) that automatically "shrinks" redundant tensor components toward zero during training. This mechanism enables the model to learn the optimal rank directly from the data, bypassing the traditional manual trial-and-error process.
Scalable Bayesian Inference
To handle large-scale datasets, the authors implement two stochastic solvers:
- Stochastic Gradient Hamiltonian Monte Carlo (SGHMC): A physics-inspired sampling method utilizing mini-batches to handle massive data.
- Stein Variational Gradient Descent (SVGD): A deterministic approach using a small set of "particles" to approximate the posterior distribution, offering a more memory-efficient alternative to HMC.
The framework is designed to be "plug-and-play," supporting various tensor formats and tasks including completion, regression, and classification.
Experimental Evaluation
The framework was evaluated on synthetic tensors, the PINCAT MRI dataset, Fashion-MNIST (for tensorized fully connected networks), and CIFAR-10 (for tensorized Convolutional Neural Networks).
Rank Recovery and Data Reconstruction
In synthetic benchmarks, the proposed methods successfully identified the ground-truth rank (5), whereas previous state-of-the-art Bayesian methods like BFCP often underestimated the rank, leading to higher error rates. On the MRI dataset, the method achieved significantly lower Root Mean Square Error (RMSE) and more accurate noise estimation than existing benchmarks.
Deep Learning Compression and Performance
The framework demonstrates significant utility in compressing deep neural networks. On the CIFAR-10 dataset, the model achieved a 127× compression ratio for a CNN while maintaining high accuracy (88.41% compared to 91.53% for the uncompressed baseline).
Key Insights
- Beyond Mean-Field Assumptions: By avoiding the restrictive independence assumptions of older methods, this framework succeeds in highly non-linear deep learning tasks where previous Bayesian tensor methods failed.
- Generalization: The Bayesian approach consistently yielded higher predictive log-likelihood (LL) and accuracy than standard Maximum A Posteriori (MAP) optimization, indicating that accounting for uncertainty helps the model generalize better to unseen data.
- Efficiency vs. Accuracy: While SVGD is faster and more memory-efficient, it tends to underestimate uncertainty compared to SGHMC.
Conclusion
This framework provides a robust solution for automated tensor learning. While Bayesian training remains more computationally intensive than standard SGD, the benefits in rank determination and uncertainty quantification are substantial for safety-critical fields. Future work will focus on reducing computational overhead and exploring more complex tensor structures.
Summary updated