Tensorized LSSVMS For Multitask Regression

Jiani Liu, Qinghua Tao, Ce Zhu, Yipeng Liu, Johan A. K. Suykens

Introduction

Multitask Learning (MTL) has emerged as a powerful paradigm in machine learning, designed to improve the generalization performance of individual tasks by exploiting the inherent "relatedness" between them. Traditionally, MTL frameworks treat tasks as a flat list, representing them via vectors or matrices. However, real-world data often possess a multi-dimensional structure that these representations fail to capture. For instance, in a recommendation system, a task might be defined by a specific customer, a specific location, and a specific rating category. Collapsing these multi-modal relationships into a simple matrix results in a significant loss of structural information.

While tensor-based methods have been proposed to address these multi-dimensional relationships, existing approaches often face two primary hurdles: the "curse of dimensionality" and a limitation to strictly linear models. This paper introduces tLSSVM-MTL, a novel framework that combines the structural richness of high-order tensor decomposition with the nonlinear modeling capabilities of Least Squares Support Vector Machines (LSSVM).

Methodology

The core innovation of the tLSSVM-MTL framework lies in its integration of tensorized weight representations with the LSSVM dual optimization problem. The methodology is built upon four pillars:

1. Tensorized Weight Representation

Instead of assigning a weight vector to each task independently, the authors construct a high-order weight tensor. Each dimension (mode) of this tensor corresponds to a different index of the task (e.g., one mode for "User," another for "Location"). This allows the model to explicitly account for the multi-way interactions between different task attributes.

2. Canonical Polyadic (CP) Decomposition

To manage the complexity of the weight tensor and avoid the curse of dimensionality, the framework factorizes the tensor using CP decomposition. This approach represents the weight tensor as a sum of rank-one tensors, effectively learning a set of shared factors that capture commonalities across all tasks alongside task-specific latent factors for each index. Unlike other methods like Tucker decomposition, CP decomposition scales linearly with the number of dimensions, ensuring computational tractability.

3. Nonlinear Kernel Extensions

By leveraging the LSSVM framework, tLSSVM-MTL supports the use of kernel functions, such as the Radial Basis Function (RBF) kernel. This enables the model to map input data into high-dimensional feature spaces, capturing complex, nonlinear relationships that traditional linear tensor models would miss.

4. Dual Optimization and Efficiency

LSSVMs utilize equality constraints rather than the inequality constraints found in standard SVMs. This choice simplifies the optimization problem into a series of linear systems. The authors employ an alternating minimization strategy (Block Coordinate Descent) to solve these systems iteratively, ensuring efficient convergence.

Experimental Evaluation

The authors evaluated tLSSVM-MTL against standard matrix-based MTL and state-of-the-art tensor methods (MLMTL-C and MLMTL-NC) across one synthetic dataset and three real-world benchmarks:

  • Comprehensive Climate Dataset (CCDS): Monthly climate records across 17 variables and 5 locations in North America.
  • Restaurant & Consumer Dataset: Ratings from 138 consumers across 3 distinct aspects.
  • Student Performance: Grade predictions across 3 time periods, segmented by demographic factors.
  • Simulated Data: A controlled environment used to test the model's robustness against varying Signal-to-Noise Ratios (SNR).

Key Results

The experimental results demonstrate the superiority of the tensorized approach, particularly when combined with nonlinear kernels. On the CCDS dataset, the proposed method achieved a $Q^2$ (coefficient of determination) of 54.50%, a substantial improvement over the standard MTL-LSSVM's 29.71%. Similarly, on the Restaurant & Consumer data, the model reached a correlation of 69.54%, outperforming the best baseline by approximately 7%.

In simulation studies, the model demonstrated high efficiency; the Root Mean Square Error (RMSE) dropped sharply and stabilized within 5 to 10 iterations, validating the effectiveness of the alternating linear system approach.

Discussion and Insights

The success of tLSSVM-MTL highlights two critical insights for the future of multitask learning. First, structure matters. By moving beyond the matrix, the model can "understand" overlapping relationships—for example, recognizing that two tasks share a user while another two share a location—and update parameters accordingly. Second, nonlinearity is essential. The "tLSSVM-MTL-rbf" variant consistently outperformed linear tensor models, suggesting that real-world task relations are rarely simple linear combinations.

Limitations and Future Work

Despite its performance gains, the model does introduce computational overhead. Training the tLSSVM variants takes significantly longer than simple matrix-based methods (e.g., ~20 seconds vs ~0.5 seconds on the Restaurant dataset). Additionally, the model requires careful tuning of the CP rank ($K$), the regularization parameter ($C$), and kernel parameters. Future research will likely focus on developing faster computation techniques and scaling these tensorized systems to massive datasets involving millions of tasks.

Summary updated

Abstract

Multitask learning (MTL) can utilize the relatedness between multiple tasks for performance improvement. The advent of multimodal data allows tasks to be referenced by multiple indices. High-order tensors are capable of providing efficient representations for such tasks, while preserving structural task-relations. In this paper, a new MTL method is proposed by leveraging low-rank tensor analysis and constructing tensorized Least Squares Support Vector Machines, namely the tLSSVM-MTL, where multilinear modelling and its nonlinear extensions can be flexibly exerted. We employ a high-order tensor for all the weights with each mode relating to an index and factorize it with CP decomposition, assigning a shared factor for all tasks and retaining task-specific latent factors along each index. Then an alternating algorithm is derived for the nonconvex optimization, where each resulting subproblem is solved by a linear system. Experimental results demonstrate promising performances of our tLSSVM-MTL.
BibTeX
@inproceedings{liu2023tensorized,
  author = {Jiani Liu and Qinghua Tao and Ce Zhu and Yipeng Liu and Johan A. K. Suykens},
  title = {{Tensorized LSSVMS For Multitask Regression}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=HbrHkxPUXy},
}
Figure