Intra-group Competition via the Exclusive Lasso

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 228 papers41 conceptsfrozen 2026-10-04

In high-dimensional machine learning, we often assume that only a small subset of features is truly relevant. This is the core of sparsity. The standard Lasso (using an $\ell_1$ norm) treats every feature as an independent competitor, while the Group Lasso (using an $\ell_{2,1}$ norm) treats them as inseparable teams—either the whole group is selected, or none of it is.

However, real-world data often possesses a more nuanced structure. Features like brain voxels, genes, or financial assets often belong to predefined categories where the members are highly redundant. In these cases, we don't want to pick every member of the team (as Group Lasso does), nor do we want to risk ignoring the entire category (as standard Lasso might). Instead, we want intra-group competition: a mechanism that forces members of the same group to compete so that only the most representative individual survives.

This is the central idea behind the Exclusive Lasso (also known as the Elitist Lasso).

The Mechanism of Competition

The Exclusive Lasso shifts the regularization strategy from "cooperation" to "competition." While the Group Lasso encourages features within a group to be selected together, the Exclusive Lasso applies a squared $\ell_1$ norm within groups.

As Deguang Kong and Ji Liu explain in their work on the Uncorrelated Group LASSO, this constraint ensures that the activation of one feature reduces the probability of others in the same group being selected. This creates an "explaining away" effect: once a representative feature has accounted for a signal, its correlated neighbors are suppressed as redundant noise.

This competitive pressure is not just a mathematical curiosity; it solves several fundamental problems in feature selection:

  1. Redundancy Filtering: In multi-label learning, Luhuan Fei and colleagues demonstrate that exclusive sparsity can eliminate up to 99% of irrelevant data interactions by forcing the model to pick only the most discriminative high-order interactions for a specific label Multi-View Multi-Label Personalized Classification via Generalized Exclusive Sparse Tensor Factorization.
  2. Representative Sparsity: Fabian L. Wauthier and Peter Donnelly show that this approach effectively handles high correlation by mathematically preventing the selection of redundant features within predefined groups A Greedy Homotopy Method for Regression with Nonconvex Constraints.

Ensuring Diversity and Representation

One of the most powerful properties of the Exclusive Lasso is its ability to ensure inter-group representation. In many applications, discarding an entire category of data is a failure.

In financial portfolio management, for instance, a model that ignores an entire economic sector increases risk. Meixia Lin, Defeng Sun, and Kim-Chuan Toh argue that exclusive sparsity is ideal here because it ensures every group contributes its most relevant members to the model A dual Newton based preconditioned proximal point algorithm for exclusive lasso models. By penalizing redundant selections within a sector rather than the sector itself, the model maintains a diversified presence across the market.

Similarly, in ensemble learning, Christopher D Manning and Mengqiu Wang used the Elitist Lasso to create a "Product of Experts." By forcing different experts to compete for the same features, the model naturally learns a "division of labor," where each expert specializes in a distinct subset of the data rather than redundantly learning the same patterns Learning a Product of Experts with Elitist Lasso.

From Global to Local Sparsity

The intuition of intra-group competition extends beyond fixed feature groups to "localized" models. In the Localized Lasso, Makoto Yamada and his team apply exclusive constraints to ensure that individual models for every data point remain active.

Standard regularizers might zero out all features for a difficult sample, effectively "giving up" on that prediction. The Exclusive Lasso prevents this collapse by inducing sparsity within the local model while ensuring the model itself remains informative. This results in extreme, interpretable sparsity where only the most relevant features for a specific individual are retained.

Computational Efficiency

Historically, the Exclusive Lasso was difficult to scale. However, recent breakthroughs in optimization have made it practical for large-scale datasets. Yancheng Yuan and colleagues developed second-order algorithms (like PPDNA) that provide closed-form solutions for the proximal mapping of the Exclusive Lasso, allowing for superlinear convergence On the Closed-form Proximal Mapping and Efficient Algorithms for Exclusive Lasso Models.

These algorithmic advances mean that the "elitist" approach—selecting the best representative from every group—is now as computationally viable as the simpler, less structured sparsity methods of the past.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 18 papers
  2. Version 1 · 2026-10-04 · 28 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_exclusive_group_sparsity_d553f762,
  title = {exclusive group sparsity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/exclusive-group-sparsity/rdr_d553f7626a7c465d8cdc72e318dc5c18},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_intra_group_competition_via_the_exclusiv_d553f762_v1,
  title = {Intra-group Competition via the Exclusive Lasso},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/intra-group-competition-via-the-exclusive-lasso/rdr_d553f7626a7c465d8cdc72e318dc5c18/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}