Abstract
This paper explores multi-task learning (MTL) in the context of available task-specific features. We propose two Gaussian Process (GP) frameworks: a "Combined Method" that integrates task descriptors into the kernel function, and a "Gating Method" based on a mixture-of-experts approach. Applied to the domain of compiler performance prediction, these methods leverage "canonical responses"—performance signatures from a small set of probe optimizations—to characterize new tasks. Our results demonstrate that the Combined Method significantly outperforms independent task learning, particularly when training data for a new program is extremely limited.
Introduction
Multi-task learning (MTL) aims to improve the generalization performance of a model by sharing information across related tasks. A significant challenge in MTL is determining how to effectively model task relationships, especially when tasks can be described by specific features or descriptors. In this work, we address the problem of predicting compiler performance. Selecting the optimal sequence of code transformations (e.g., loop unrolling, inlining) for a specific program is a high-dimensional search problem. We aim to use transfer learning to predict the performance of a new, unseen program by leveraging data from previously analyzed programs.
Methodology
We propose two GP-based approaches for MTL that utilize task-specific features $f$:
- The Combined Method: This approach models the joint distribution of all tasks. The input vector is formed by concatenating the transformation sequence $x$ with the program descriptors $f$. We employ a product kernel $k((x, f), (x', f')) = k_x(x, x') \cdot k_f(f, f')$. This structure allows the model to automatically infer task correlations; programs with similar descriptors are expected to exhibit similar performance responses to the same transformations.
- The Gating Method: This approach utilizes a mixture-of-experts framework. A separate GP model is trained for each reference program. For a new program, a gating network uses the descriptors $f$ to weight the predictions of these existing "experts," effectively blending knowledge from the most relevant source tasks.
A critical component is the choice of task features $f$. We found that "canonical responses"—the speed-up achieved on a small, fixed set of "principal" transformation sequences—serve as a more effective signature for a program's performance characteristics than static source code features.
Experimental Results
We evaluated our methods using the UTDSP benchmark suite on a Texas Instruments C6713 DSP board, exploring a search space of over 88,000 transformation sequences.
As shown in Figure 2, the Combined Method using transformation-based features consistently achieved the lowest Mean Absolute Error (MAE). Across all benchmarks, it reached an average MAE of 0.0576, a substantial improvement over the baseline median predictor (MAE of 0.1162). The Gating Method, while better than independent learning, was generally less effective than the Combined Method, likely due to the independent training of its expert components.
The "Cold Start" Advantage
The primary benefit of transfer learning is observed when data for the target task is scarce. Figure 3 illustrates the learning curves for various benchmarks, comparing the Combined Method against models trained without transfer learning.
The transfer learning models provide a significant accuracy boost when only 8 to 16 test runs are available for the new program. As the number of task-specific training samples increases to 128, the performance gap narrows, as the model can increasingly rely on direct observations from the target task. This "cold start" capability is vital for practical compiler optimization, where running thousands of trials is computationally prohibitive.
Conclusion
We have demonstrated that Gaussian Processes provide a principled and effective framework for multi-task learning using task-specific descriptors. By using canonical responses as program signatures, our Combined Method accurately predicts performance for unseen programs with minimal experimental overhead. Future work will investigate joint training for the Gating Method and more complex, non-factorized kernel structures to further enhance predictive accuracy.
Summary updated