Gradient Boosting for Efficient Rank Discovery in Tensor Network Kernel Machines

Albert Saiapin, Kim BatselierarXiv 2026 · arXiv

TL;DR

Researchers have developed GB-CPKM, a gradient boosting framework that eliminates the need for expensive grid searches to find the optimal rank in tensor network models. By sequentially adding low-rank "base learners" to an ensemble, the method achieves up to 6x faster training and 2x lower peak memory usage compared to traditional cross-validation, without sacrificing predictive accuracy.

What Problem Does This Paper Address?

Tensor Network Kernel Machines (TNKMs) are powerful tools for modeling complex, high-dimensional data. They avoid the "curse of dimensionality" by representing massive weight vectors as a network of smaller, low-rank tensors—specifically using the Canonical Polyadic (CP) decomposition.

However, these models have a major Achilles' heel: Rank Selection. The "rank" ($R$) determines the model's capacity. If the rank is too low, the model underfits; if it's too high, it overfits and becomes computationally bloated. Currently, practitioners find the right rank through trial and error (cross-validation), training multiple models from scratch (e.g., Rank 2, then Rank 4, then Rank 6...). This is incredibly slow and memory-intensive, especially for large-scale datasets.

Methods

The authors propose GB-CPKM (Gradient Boosting for CP Kernel Machines). Instead of training one large model with a fixed rank, they apply the principles of Gradient Boosting:

  1. Sequential Growth: Start with a simple model.
  2. Residual Fitting: Train a new, very low-rank "base learner" (e.g., rank $\hat{R}=2$) to predict the errors (residuals) of the current ensemble.
  3. Additive Ensemble: Add this new learner to the ensemble. Because of a unique property of CP tensors, the sum of two CP tensors is simply a new CP tensor with a rank equal to the sum of their individual ranks.
  4. Early Stopping: Stop adding learners when performance on a validation set plateaus.

This transforms rank selection from a discrete search problem into a continuous growth process. They also introduce a hybrid version, GB-CPKM(H), which performs one final optimization sweep over the entire combined model to squeeze out extra accuracy.

Comparison of the training procedures: GB-CPKM grows the model incrementally, while CV-CPKM trains multiple models from scratch.

Datasets

The authors evaluated their method on seven real-world regression benchmarks:

  • Small/Medium Scale: Energy, Concrete, Naval, and HouseElectric (from the UCI repository).
  • Large Scale: Kin8nm, Precipitation (628k samples), and Airline (5.9 million samples).

Key Insights and Ideas

  • The "Sum" Property: The core "aha!" moment is leveraging the fact that adding CP tensors is mathematically equivalent to concatenating their factor matrices. This allows a boosted ensemble to be represented as a single, standard CP model.
  • Memory Efficiency: In traditional cross-validation, peak memory is dictated by the largest rank you test. In GB-CPKM, peak memory is largely determined by the rank of the base learner, which stays small even as the total ensemble rank grows.
  • Base Learner Rank Matters: The authors found that using slightly higher-rank base learners (e.g., rank 4 or 8) is often more efficient than using many rank-1 learners. Higher-rank learners capture more complex patterns in fewer boosting steps.
Figure 2: Ablation study of base-learner rank in GB-CPKM on four real-world datasets, with each column corresponding to a dataset. The top row shows cumulative training time (in seconds), while the bottom row reports validation MSE on a logarithmic scale. Across the evaluated datasets, higher-rank base learners achieve better predictive performance while also reducing cumulative training time. However, rank selection should also account for memory requirements, which may constrain the use of higher-rank models in practice.

Key Results

  • Speed: GB-CPKM is significantly faster. On the HouseElectric dataset, it trained in ~25 seconds compared to ~270 seconds for the cross-validation baseline—a 10x speedup in that specific case (averaging 6x across all tests).
  • Memory: Peak memory usage was halved on large datasets like Airline, dropping from nearly 10GB to roughly 4GB.
  • Accuracy: The predictive performance (MSE) remained comparable to the much more expensive cross-validation approach. The hybrid variant, GB-CPKM(H), often matched or exceeded the baseline accuracy.
Performance metrics showing GB-CPKM (red) scales much more gracefully in time and memory than CV-CPKM (blue).

Limitations and Future Work

While highly efficient, the sequential nature of boosting means you cannot easily parallelize the training of different boosting stages (unlike cross-validation, where you could train Rank 2 and Rank 10 simultaneously on different machines).

The authors suggest several future paths:

  • Extending the framework to Tensor Train (TT) decompositions, which are more stable than CP but have more complex rank parameters.
  • Applying the method to classification and time-series forecasting.
  • Investigating "rank pruning" strategies where the rank could decrease if a boosting stage is found to be redundant.

Summary updated

Abstract

Tensor network kernel machines (TNKMs) combine nonlinear features with low-rank tensor representations to model complex relationships efficiently. However, selecting an appropriate tensor rank remains challenging, as conventional rank-search methods require training multiple candidate models, increasing computational costs. In this work, we focus on a canonical polyadic (CP) tensor network parameterization and propose GB-CPKM, a gradient boosting framework that builds a higher-capacity predictor by sequentially adding fixed-rank learners. This approach provides a new perspective on rank selection by incrementally increasing model capacity, with ensemble size selected based on validation performance rather than through explicit rank search. Experiments on real-world regression benchmarks show that GB-CPKM trains up to 6x faster and uses up to 2x less peak memory than a conventional rank-selection method based on cross-validation, while achieving comparable predictive performance. These results highlight the potential of gradient boosting to improve the computational efficiency and scalability of tensor-network learning.
BibTeX
@misc{saiapin2026gradient,
  author = {Albert Saiapin and Kim Batselier},
  title = {{Gradient Boosting for Efficient Rank Discovery in Tensor Network Kernel Machines}},
  year = {2026},
  eprint = {2611.08093},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08093},
}
Figure