Abstract
Multi-task learning seeks to improve the generalization performance of a learning task with the help of other related learning tasks. Among the multi-task learning methods proposed thus far, Bonilla et al.’s method provides a novel multi-task extension of Gaussian process (GP) by using a task covariance matrix to model the relationships between tasks. However, learning the task covariance matrix directly has both computational and representational drawbacks. In this paper, we propose a Bayesian extension by modeling the task covariance matrix as a random matrix with an inverse-Wishart prior and integrating it out to achieve Bayesian model averaging. To make the computation feasible, we first give an alternative weight-space view of Bonilla et al.’s multi-task GP model and then integrate out the task covariance matrix in the model, leading to a multi-task generalized t process (MTGTP). For the likelihood, we use a generalized t noise model which, together with the generalized t process prior, brings about the robustness advantage as well as an analytical form for the marginal likelihood. In order to specify the inverse-Wishart prior, we use the maximum mean discrepancy (MMD) statistic to estimate the parameter matrix of the inverse-Wishart prior. Moreover, we investigate some theoretical properties of MTGTP, such as its asymptotic analysis and learning curve. Comparative experimental studies on two common multi-task learning applications show very promising results.
Introduction
Multi-task learning (MTL) is based on the principle that learning related tasks simultaneously—such as predicting exam scores across different schools—is more effective than learning them in isolation. A major milestone in this field was the Multi-Task Gaussian Process (MTGP), which utilizes a "task covariance matrix" to model inter-task correlations. However, the standard MTGP faces two significant challenges. First, direct optimization of the task covariance matrix is non-convex, often leading to suboptimal local optima. Second, Gaussian Processes are highly sensitive to outliers; the assumption of normally distributed noise causes the model to overfit extreme data points, which can degrade overall predictive performance.
Methodology
To address these limitations, we propose the Multi-Task Generalized $t$ Process (MTGTP). This framework incorporates three key innovations:
- Bayesian Model Averaging: Rather than estimating a fixed task covariance matrix, we treat it as a random variable following an inverse-Wishart prior. Integrating this matrix out provides a more stable model that avoids the pitfalls of point estimation.
- Transition to $t$-Processes: The integration of a Gaussian likelihood with an inverse-Wishart prior mathematically transforms the model into a Generalized $t$ Process. The resulting heavy-tailed distribution naturally accommodates outliers, enhancing model robustness.
- MMD-based Initialization: We employ the Maximum Mean Discrepancy (MMD) statistic to provide a non-parametric estimate of task similarities, serving as an informed starting point for the inverse-Wishart prior.
By adopting a weight-space view, we derive an analytical form for the marginal likelihood. This ensures that the model remains computationally efficient and compatible with standard gradient-based optimization techniques, avoiding the need for slow approximate simulations. Furthermore, the paper provides theoretical grounding through asymptotic analysis and learning curves, proving predictable error convergence as data volume increases.
Experimental Results
The MTGTP model was evaluated against single-task GPs, Multi-Task Feature Learning (MTFL), and the original MTGP on two classic multi-task regression benchmarks.
London School Data
In the London School dataset, which involves predicting exam scores for over 15,000 students across 139 schools, MTGTP consistently outperformed competing methods. As shown in Table 1, with a 30% training set, MTGTP achieved a normalized Mean Squared Error (nMSE) of 0.6412, a significant improvement over the 0.8094 achieved by the standard Multi-Task GP.
SARCOS Robot Arm
The performance gains were even more dramatic in the SARCOS robot arm dynamics task. For a training set of 500 points, MTGTP reached an nMSE of 0.0138, whereas the standard Multi-Task GP lagged significantly at 0.2453. This demonstrates the model's superior ability to handle complex, multi-dimensional regression tasks where precision is critical.
Discussion and Conclusion
MTGTP provides a robust, Bayesian alternative to standard multi-task GPs. By bridging the gap between Bayesian model averaging and $t$-process robustness, it achieves state-of-the-art results on complex regression benchmarks.
Despite its success, the model's $O(n^3)$ computational complexity remains a challenge for large-scale datasets, though approximations like the Nyström method offer a potential path forward. Additionally, the current formulation assumes shared degrees of freedom between the prior and noise model to maintain analytical tractability. Future research could explore relaxing this constraint or extending the framework to multi-task classification tasks using a probit likelihood.
Summary updated