Abstract: Tensor train (TT) decomposition, a powerful tool for analyzing multidimensional data, exhibits superior performance in many machine learning tasks. However, existing methods for TT decomposition either suffer from noise overfitting, or require extensive fine-tuning of the balance between model complexity and representation accuracy. In this paper, a fully Bayesian treatment of TT decomposition is employed to avoid noise overfitting, by endowing it with the ability of automatic rank determination. In particular, theoretical evidence is established for adopting a Gaussian-product-Gamma prior to induce sparsity on the slices of the TT cores, so that the model complexity is automatically determined even under incomplete and noisy observed data. Furthermore, based on the proposed probabilistic model, an efficient learning algorithm is derived under the variational inference framework. Simulation results on synthetic data show the success of the proposed model and algorithm in recovering the ground-truth TT structure from incomplete noisy data. Further experiments on real-world data demonstrate the proposed algorithm performs better in image completion and image classification, compared to other existing TT decomposition algorithms.
Introduction
Tensors—multidimensional arrays—are the natural format for complex data structures like video, hyperspectral images, and neural network weights. Tensor Train (TT) decomposition has emerged as a popular method to compress these massive arrays into smaller, manageable "cores." However, traditional optimization-based TT methods face two significant hurdles: overfitting in the presence of noise and the difficulty of manual rank estimation. Researchers often have to guess the optimal rank or spend significant effort fine-tuning regularization parameters to balance accuracy and compression.
This paper introduces a robust Bayesian framework designed to address these challenges automatically. By moving toward probabilistic modeling, the authors provide a "set-it-and-forget-it" approach that handles noise gracefully while discovering the true underlying rank of the data.
Methodology
The core innovation of this work is the introduction of a Gaussian-product-Gamma prior. In a Tensor Train, adjacent cores are coupled through their shared ranks. To exploit this, the authors model the precision (inverse variance) of the core slices as a product of two Gamma-distributed variables.
Sparsity Induction and Rank Determination
The authors provide formal theoretical proof that this specific prior creates a "heavy-tailed" distribution with a sharp peak at zero. This encourages the model to "prune" unnecessary slices of the tensor cores during the learning process. If a slice is not required to explain the observed data, the Bayesian prior pushes its values toward zero, effectively discovering the optimal rank automatically.
Variational Inference (VI)
To make the model computationally tractable, the authors derive a Mean-Field Variational Inference algorithm. This approach iteratively updates the model's beliefs about the cores and the noise level until convergence. Furthermore, they derived a mathematical optimization to swap the order of summations and Kronecker products, significantly accelerating the algorithm for fully observed tensors.
Experimental Evaluation
The model was rigorously evaluated across synthetic and real-world datasets:
- Synthetic Data: On order-3 tensors with known ground-truth, the model achieved 100% accuracy in recovering true ranks at 20dB SNR, even with 20% missing data.
- Image Completion: Testing on 13 standard benchmark images (e.g., 'airplane', 'lena') with 80% pixel removal, the proposed method consistently outperformed competitors. For the 'airplane' image under Gaussian noise, it achieved a PSNR of 18.51 dB, compared to the next best competitor's 16.57 dB.
- Hyperspectral Images (HSI): Using the CAVE dataset, the model demonstrated superior recovery under structured "stripe" missing patterns.
- Image Classification: On the CIFAR-10 dataset, the Bayesian TT approach maintained lower error rates under noisy conditions than standard TT-SVD, proving its ability to capture true underlying structures rather than noise.
Conclusion
This research provides a mathematically grounded framework for automatic rank determination in Tensor Train decomposition. By treating the noise level as a learnable variable and employing a sparsity-inducing prior, the model avoids the pitfalls of manual tuning and overfitting. While the computational complexity of $O(DL^4)$ remains a consideration for massive datasets, the framework sets a new standard for robust multidimensional data analysis. Future work may extend this prior analysis to other formats like Tensor Ring or non-Gaussian noise models.
Summary updated