Abstract
We empirically investigate the best trade-off between sparse and uniformly-weighted multiple kernel learning (MKL) using the elastic-net regularization on real and simulated datasets. We find that the best trade-off parameter depends not only on the sparsity of the true kernel-weight spectrum but also on the linear dependence among kernels and the number of samples.
Introduction: The Sparse vs. Uniform Dilemma
Multiple Kernel Learning (MKL) is a powerful framework for learning optimal combinations of kernels, allowing models to integrate diverse data sources or feature representations. Historically, the field has been divided between two extremes: sparse MKL (using $\ell_1$-norm regularization), which promotes interpretability and computational efficiency by selecting a few kernels, and uniform MKL (using $\ell_2$-norm), which averages kernels equally. While sparse models are theoretically appealing, practitioners often find that uniform weighting yields superior accuracy in real-world applications. This paper investigates the "sparsity-accuracy trade-off" to determine if a middle ground exists and what factors influence the optimal balance.
Methodology: Bridging the Gap with Elastic-Net
To explore the continuum between sparse and uniform models, the authors employ elastic-net regularization. This approach introduces a trade-off parameter, $\lambda$, where $\lambda = 0$ corresponds to purely sparse $\ell_1$-MKL and $\lambda = 1$ corresponds to uniform $\ell_2$-MKL. By adjusting $\lambda$, the model can blend these strategies to find a "sweet spot."
To handle the computational demands of testing thousands of kernels, the researchers extended the SpicyMKL optimization algorithm. This extension allows for efficient training even when the regularization is non-sparse, making it possible to evaluate large-scale kernel libraries across various settings.
Experimental Setup
The study utilizes both real-world and synthetic benchmarks:
- Caltech 101: 10 binary classification tasks using a library of 1,760 kernels per task, derived from various SIFT features and spatial pyramids.
- Simulated Data: Three scenarios designed to isolate variables: Feature Selection (independent kernels), Parameter Selection (dependent kernels with varying bandwidths), and a hybrid Feature & Parameter Selection.
Results and Discussion: The Dynamics of the Sweet Spot
Real-World Performance on Caltech 101
The experiments on Caltech 101 reveal a clear relationship between sample size ($N$) and the optimal regularization strategy. When training data is scarce ($N=5$), uniform weighting ($\lambda \approx 1$) is significantly more robust. As the number of samples increases to $N=30$, the optimal solution shifts toward lower $\lambda$ values, indicating that the model can more accurately identify a sparse subset of essential kernels.
Simulation Trends and Kernel Dependency
The simulation results highlight the impact of kernel relationships. In scenarios where kernels are independent, sparse MKL performs well even with limited data. However, in more realistic settings where kernels exhibit linear dependencies (e.g., similar bandwidths), the elastic-net approach consistently outperforms both extremes. The "Dependency Trap" in sparse MKL—where the model arbitrarily discards highly correlated but useful kernels—is mitigated by the elastic-net's ability to retain groups of similar kernels.
Conclusion
The optimal trade-off in MKL is not static; it is a function of sample size, kernel dependency, and the underlying sparsity of the true signal. Uniform weighting acts as a necessary regularizer in small-data regimes, while sparsity becomes more advantageous as data volume grows. Elastic-net MKL provides a flexible framework to navigate this trade-off, offering a robust path to maximizing predictive accuracy across diverse learning scenarios.
Summary updated