Abstract
This paper introduces RMKL (Multi-View Randomized Kernel Classification), a novel framework designed to address the diversity-accuracy tradeoff in Multiple Kernel Learning (MKL). By generating a pool of high-quality randomized kernels and selecting an optimal subset through nonconvex optimization, RMKL achieves state-of-the-art performance. The method transforms a difficult integer programming problem into a solvable semidefinite relaxation, demonstrating robustness across diverse datasets and various kernel-based classifiers.
Introduction
Multi-view data integration—combining information from sources like text, images, and audio—is a cornerstone of modern machine learning. Multiple Kernel Learning (MKL) is a primary technique for this, yet it often struggles with the balance between kernel accuracy and diversity. Traditional methods frequently overlook the importance of diversity, leading to redundant kernels that fail to improve generalization. Furthermore, standard kernels are highly sensitive to hyperparameter tuning, making the search for optimal parameters computationally expensive and prone to sub-optimal results.
Methods
The proposed RMKL framework operates in three distinct stages:
- Generating Randomized Kernels: Instead of manual tuning, random parameters are assigned to each input dimension. This creates a large pool of randomized kernels capable of capturing diverse characteristics of the data distribution.
- Measuring Quality: Each kernel is evaluated based on a "double-fault" diversity metric (how often two kernels fail on the same sample) and individual accuracy via $k$-fold cross-validation.
- Nonconvex Optimization: The selection of the optimal kernel subset is formulated as a nonconvex quadratic integer programming problem. This is efficiently solved by transforming it into a convex semidefinite programming (SDP) relaxation and applying a branch-and-bound (B&B) algorithm.
Results
RMKL was evaluated on eight benchmark datasets, including four multi-view datasets (e.g., 3Sources, MSRC-v1) and four single-view gene expression datasets (e.g., Armstrong, Chen). The results demonstrate that RMKL consistently achieves superior accuracy, outperforming ten state-of-the-art MKL methods.
As shown in Figure 1, the method is classifier-agnostic, providing performance gains not only for standard Support Vector Machines (SVM) but also for Least Squares SVM (LSSVM) and Least Squares Kernel Extreme Learning Machines (LSKELM).
Sensitivity analysis (Figure 4) further reveals that RMKL is relatively robust to the choice of hyperparameters, such as the number of selected kernels ($m$) and the regularization parameter ($C$), facilitating its application in real-world scenarios.
Conclusion
RMKL provides a mathematically elegant solution to the MKL diversity-accuracy tradeoff. Its use of randomization reduces parameter sensitivity, while the optimization framework ensures a diverse and accurate kernel subset. While the computational complexity of solving SDPs remains a consideration for very large kernel pools, the current results establish RMKL as a robust and high-performing framework for multi-view learning. Future work will involve extending this approach to regression and clustering tasks.
Summary updated