Abstract
Kernel methods are robust for non-linear modeling but face significant scalability challenges due to cubic computational complexity. While mapping data into high-dimensional feature spaces using tensor-product structures offers a solution, it introduces the "curse of dimensionality" and necessitates exhaustive hyperparameter tuning via cross-validation (CV). This paper introduces a Feature Learning (FL) model that treats hyperparameter selection as a learnable tensor decomposition problem. By utilizing a Canonical Polyadic Decomposition (CPD) framework and an Alternating Least Squares (ALS) optimization strategy, the FL model automates feature selection during training. Experimental results on datasets ranging from small-scale regression to the 5.9-million-sample Airline dataset demonstrate that the FL model achieves 3–5x faster training times than CV while maintaining competitive predictive accuracy.
Introduction
Kernel methods, such as Support Vector Machines and Gaussian Processes, are powerful tools for non-linear modeling. However, their $O(N^3)$ complexity relative to dataset size $N$ limits their application to large-scale problems. Primal formulations using tensor-product features (e.g., Fourier or polynomial bases) mitigate this, but they require careful selection of hyperparameters—such as basis frequencies—which is typically performed through computationally expensive cross-validation. This process is often the primary bottleneck in deploying kernel-based models on massive datasets.
Methods: Feature Learning via Tensor Decomposition
The proposed Feature Learning (FL) model integrates hyperparameter selection directly into the training process. Instead of selecting a single fixed feature map, the model considers a linear combination of $P$ candidate feature maps. This is represented through a Canonical Polyadic Decomposition (CPD), a tensor network technique that decomposes complex multidimensional arrays into simpler components.
Optimization is performed using Alternating Least Squares (ALS), which iteratively updates feature importance weights ($\lambda$) and model parameters ($w$). This approach transforms a non-linear optimization problem into a sequence of linear sub-problems. Furthermore, the application of L1 regularization to the feature weights induces sparsity, effectively "turning off" irrelevant features. This sparsity provides a literal speed boost during training by removing those features' contributions from subsequent heavy computations. To further enhance performance, the model utilizes quantized Fourier features, allowing for logarithmic scaling with respect to feature dimensionality.
Experimental Results
The FL model was evaluated against a cross-validation baseline on several UCI regression datasets (Airfoil, Concrete, Energy, Wine, Yacht) and the large-scale Airline dataset containing 5.9 million records.
As shown in the results, the FL model consistently outperformed CV in terms of training speed, offering a 3x to 5x speedup across all benchmarks. For the Airline dataset, training time was reduced from 15.7 hours to approximately 4.2 hours. Crucially, the Mean Squared Error (MSE) remained competitive with the CV baseline. In some instances, such as the Yacht dataset, the FL model actually outperformed CV, likely because the learned combination of multiple features is more expressive than any single feature set chosen by standard validation methods.
Conclusion
By reframing hyperparameter tuning as a differentiable weight-learning problem within a tensor network framework, the FL model significantly enhances the efficiency of kernel-based learning. While the model currently requires an initial pool of candidate hyperparameters, it eliminates the need for exhaustive search. Future work will explore parallelization strategies to leverage the model's inherent summation structure and extension to classification and probabilistic modeling tasks.
Summary updated