Abstract
This paper introduces Multi-Layer Multiple Kernel Learning (MLMKL), a framework that moves beyond traditional "shallow" kernel combinations to a hierarchical, deep architecture. By stacking kernel evaluations in two layers (2LMKL), the authors provide a flexible model that captures complex data patterns more effectively than standard Multiple Kernel Learning (MKL) or single-kernel SVMs. Empirical evaluations across 16 benchmark datasets demonstrate that 2LMKL-Inf consistently achieves superior accuracy and ranking compared to state-of-the-art baselines.
Introduction
Kernel methods, such as Support Vector Machines (SVMs), are fundamental to machine learning due to their ability to map data into high-dimensional spaces for linear separation. However, their performance is heavily dependent on the choice of the kernel function. Traditional Multiple Kernel Learning (MKL) attempts to automate this by learning a linear or convex combination of predefined base kernels.
Despite its theoretical appeal, MKL often fails to outperform a single, well-tuned RBF kernel in practice. The authors argue that this is because MKL is "shallow"—it flattens multiple kernels into a single representation that lacks the richness to capture high-order interactions. This research aims to bridge the gap between kernel methods and the hierarchical representational power characteristic of deep learning.
Methodology
The proposed framework organizes kernels into layers, mimicking the structure of a neural network to enhance representational capacity.
Two-Layer MKL (2LMKL)
In the 2LMKL model, the second-layer kernel is defined as an exponential combination of first-layer kernels: $$k^{(2)}(x_i, x_j; \mu) = \exp\left(\sum_{t=1}^m \mu_t k_t^{(1)}(x_i, x_j)\right)$$ This non-linear combination allows the model to capture complex synergies between base kernels that a simple sum would miss.
Optimization and Automation
Learning the model involves a min-max optimization problem solved via an alternating strategy: fixing kernel weights ($\mu$) to solve for SVM dual variables ($\alpha$), and then updating $\mu$ using gradient ascent. To further automate the process, the authors developed 2LMKL-Inf, which iteratively "grows" the first layer by searching for optimal kernel parameters (e.g., Gaussian bandwidth), effectively automating feature engineering.
Experimental Setup
The researchers evaluated the approach on 16 binary classification benchmark datasets from the UCI repository and LibSVM. These datasets span various domains, including medical (Breast Cancer, Diabetes), physical signals (Ionosphere, Sonar), and social data (Adult, German Credit). The diversity of these benchmarks ensures a robust assessment of the model's generalizability.
Key Results
The authors compared 2LMKL and 2LMKL-Inf against seven other methods, including standard SVMs and state-of-the-art MKL variants like LpMKL, GMKL, and IKL.
- Superior Accuracy: 2LMKL-Inf achieved the highest accuracy in 11 out of 16 datasets.
- Competitive Ranking: 2LMKL-Inf obtained an average rank of 2.13, significantly outperforming the best "shallow" MKL method (IKL at 4.00) and the previous deep kernel method (MKM at 4.00).
- Consistency: Unlike traditional MKL, which can sometimes underperform a single SVM, the two-layer methods consistently matched or improved upon the baseline SVM performance.
Discussion and Conclusion
A significant theoretical contribution of this work is proving that the "pseudo-dimension" of the two-layer kernel remains bounded by the number of base kernels ($m$). This ensures that the model gains flexibility without a proportional increase in the risk of overfitting.
While the current alternating optimization is more computationally intensive than standard MKL, the results suggest that hierarchical kernel stacking is a viable path toward more powerful kernel methods. Future work may explore deeper architectures (three or more layers) and layer-wise training techniques to further improve efficiency and representational depth.
Summary updated