Tensor Network Based Feature Learning Model

Albert Saiapin, kim batselier

Abstract

Kernel methods are robust for non-linear modeling but face significant scalability challenges due to cubic computational complexity. While mapping data into high-dimensional feature spaces using tensor-product structures offers a solution, it introduces the "curse of dimensionality" and necessitates exhaustive hyperparameter tuning via cross-validation (CV). This paper introduces a Feature Learning (FL) model that treats hyperparameter selection as a learnable tensor decomposition problem. By utilizing a Canonical Polyadic Decomposition (CPD) framework and an Alternating Least Squares (ALS) optimization strategy, the FL model automates feature selection during training. Experimental results on datasets ranging from small-scale regression to the 5.9-million-sample Airline dataset demonstrate that the FL model achieves 3–5x faster training times than CV while maintaining competitive predictive accuracy.

Introduction

Kernel methods, such as Support Vector Machines and Gaussian Processes, are powerful tools for non-linear modeling. However, their $O(N^3)$ complexity relative to dataset size $N$ limits their application to large-scale problems. Primal formulations using tensor-product features (e.g., Fourier or polynomial bases) mitigate this, but they require careful selection of hyperparameters—such as basis frequencies—which is typically performed through computationally expensive cross-validation. This process is often the primary bottleneck in deploying kernel-based models on massive datasets.

Methods: Feature Learning via Tensor Decomposition

The proposed Feature Learning (FL) model integrates hyperparameter selection directly into the training process. Instead of selecting a single fixed feature map, the model considers a linear combination of $P$ candidate feature maps. This is represented through a Canonical Polyadic Decomposition (CPD), a tensor network technique that decomposes complex multidimensional arrays into simpler components.

Comparison of the standard Tensorized CPD Kernel Machine (a) and the proposed Feature Learning Model (b).

Optimization is performed using Alternating Least Squares (ALS), which iteratively updates feature importance weights ($\lambda$) and model parameters ($w$). This approach transforms a non-linear optimization problem into a sequence of linear sub-problems. Furthermore, the application of L1 regularization to the feature weights induces sparsity, effectively "turning off" irrelevant features. This sparsity provides a literal speed boost during training by removing those features' contributions from subsequent heavy computations. To further enhance performance, the model utilizes quantized Fourier features, allowing for logarithmic scaling with respect to feature dimensionality.

Experimental Results

The FL model was evaluated against a cross-validation baseline on several UCI regression datasets (Airfoil, Concrete, Energy, Wine, Yacht) and the large-scale Airline dataset containing 5.9 million records.

Performance comparison of the Feature Learning (FL) model versus Cross-Validation (CV) across small and large-scale datasets.

As shown in the results, the FL model consistently outperformed CV in terms of training speed, offering a 3x to 5x speedup across all benchmarks. For the Airline dataset, training time was reduced from 15.7 hours to approximately 4.2 hours. Crucially, the Mean Squared Error (MSE) remained competitive with the CV baseline. In some instances, such as the Yacht dataset, the FL model actually outperformed CV, likely because the learned combination of multiple features is more expressive than any single feature set chosen by standard validation methods.

Conclusion

By reframing hyperparameter tuning as a differentiable weight-learning problem within a tensor network framework, the FL model significantly enhances the efficiency of kernel-based learning. While the model currently requires an initial pool of candidate hyperparameters, it eliminates the need for exhaustive search. Future work will explore parallelization strategies to leverage the model's inherent summation structure and extension to classification and probabilistic modeling tasks.

Summary updated

Abstract

Many approximations were suggested to circumvent the cubic complexity of kernel-based algorithms, allowing their application to large-scale datasets. One strategy is to consider the primal formulation of the learning problem by mapping the data to a higher-dimensional space using tensor-product structured polynomial and Fourier features. The curse of dimensionality due to these tensor-product features was effectively solved by a tensor network reparameterization of the model parameters. However, another important aspect of model training — identifying optimal feature hyperparameters — has not been addressed and is typically handled using the standard cross-validation approach. In this paper, we introduce the Feature Learning (FL) model, which addresses this issue by representing tensor-product features as a learnable Canonical Polyadic Decomposition (CPD). By leveraging this CPD structure, we efficiently learn the hyperparameters associated with different features alongside the model parameters using an Alternating Least Squares (ALS) optimization method. We prove the effectiveness of the FL model through experiments on real data of various dimensionality and scale. The results show that the FL model can be consistently trained 3-5 times faster than and have the prediction quality on par with a standard cross-validated model.
BibTeX
@inproceedings{saiapin2025tensor,
  author = {Albert Saiapin and kim batselier},
  title = {{Tensor Network Based Feature Learning Model}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=Dttmp0TPVY},
}
Figure