In machine learning, the principle of parsimony often dictates that simpler models are better. This intuition is frequently codified through $\ell_1$ regularization (like the Lasso), which encourages "sparsity"—the idea that a model should rely on only a handful of the most important features or kernels while zeroing out the rest.
However, a growing body of research suggests that in complex, real-world domains, this "winner-takes-all" approach is a liability. When predictive signals are subtle, distributed, or complementary, aggressive sparsity doesn't just simplify the model; it blinds it.
The Sparsity Trap
The traditional justification for sparse Multiple Kernel Learning (MKL) is that it performs automatic feature selection. If you have 50 different ways to represent an image (color histograms, texture maps, shape descriptors), a sparse model will pick the "best" two or three.
As Alexander Binder and Motoaki Kawanabe observed in their work on visual concept classification, this aggressive pruning often discards kernels that contain unique, non-redundant information. In tasks like image annotation, a "weak" feature might be the only one capable of distinguishing a specific edge case. When $\ell_1$ regularization zeros it out, that nuance is lost forever.
This phenomenon is often called the "curse of dimensionality" in reverse. Sathiya Keerthi noted in text classification that relying solely on $\ell_1$ can be too aggressive, removing features that contribute to the overall robustness of the model.
From Sparsity to Density
The alternative is non-sparse regularization, typically using $\ell_p$-norms where $p > 1$. While an $\ell_1$ norm forces the model onto the corners of a simplex (selecting one kernel and discarding others), an $\ell_2$ or $\ell_p$ norm allows for a "dense" combination where many kernels contribute.
Ulf Brefeld, Marius Kloft, and colleagues demonstrated in Insights from Classifying Visual Concepts that non-sparse MKL consistently outperforms sparse versions. They argued that the "humble average" of kernels is often a harder baseline to beat than people realize, and non-sparse MKL provides a principled way to learn weights that are better than a simple average without falling into the trap of over-sparsity.
The theoretical foundation for this was furthered in Fast Learning Rate of Non-Sparse Multiple Kernel Learning, which proved that dense regularization is mathematically superior when the candidate kernels have varying levels of complexity—a standard reality in data science.
Why Density Works: Complementary Information
The core intuition for why dense models win is complementarity. In biological and medical data, a single "strong" biomarker rarely tells the whole story.
- Metagenomics: Gregory Ditzler and Gail Rosen found that Lasso-style methods might reduce a complex disease phenotype to a single microbial taxon. This is mathematically efficient but biologically incomplete, as it ignores the diverse ecosystem of bacteria that collectively drive a condition.
- Alzheimer’s Research: When classifying brain scans, Fayao Liu and Chunhua Shen showed that sparse penalties discard kernels representing different brain regions that provide subtle, additive signals.
- Protein Folding: Kaizhu Huang and Colin Campbell used an information-theoretic approach to integrate heterogeneous data, finding that retaining "secondary" data sources was critical for state-of-the-art accuracy.
Overcoming the Computational Hurdle
If non-sparse models are better, why did the field obsess over sparsity for so long? The answer is largely computational. Sparse models are often easier to optimize because they effectively reduce the size of the problem as they go.
Recent work has bridged this gap. Zenglin Xu and Irwin King developed Smooth Multiple Kernel Learning (SMKL), which uses entropy regularization to make the optimization landscape easier to navigate. Similarly, Xu showed that MKL can be reframed as a Group Lasso problem, allowing for closed-form updates that make training non-sparse models nearly as fast as their sparse counterparts.
The Bayesian Perspective
The tension between sparsity and density also appears in Bayesian modeling. Ryota Tomioka and Taiji Suzuki demonstrated that empirical Bayesian learning can automatically tune these weights. They found that in low-sample regimes, sparse models are particularly dangerous because they might "over-prune" based on noise, whereas a denser, regularized approach maintains a more robust "collective strength" across features.
Ultimately, the shift toward non-sparse regularization represents a move away from the "needle in a haystack" philosophy toward a "strength in numbers" approach. By allowing models to be dense, researchers like Corinna Cortes and Mehryar Mohri have shown that we can build more stable and accurate systems that respect the inherent complexity of the data they are trying to model.