In high-dimensional data analysis, unsupervised learning faces a classic chicken-and-egg dilemma. To group data into meaningful clusters, we need to know which features are relevant; noisy or irrelevant variables can easily swamp the true underlying signal. Conversely, to determine which features are truly informative, we need to understand the underlying group structure of the data.
Historically, practitioners bypassed this dilemma using a sequential pipeline: first, rank and select features using an unsupervised metric (such as variance or Laplacian score), and second, run a clustering algorithm on the reduced dataset. However, as Jun Guo 0008 et al. demonstrate in Dependence Guided Unsupervised Feature Selection, this decoupled approach is fundamentally sub-optimal. Features selected in isolation have no guarantee of aligning with the specific geometric or structural partitions that a clustering algorithm seeks to uncover.
The core insight of joint feature selection and clustering is to dissolve the boundary between these two steps, treating them as a single, unified optimization problem. By co-optimizing both tasks, the model establishes a virtuous cycle: the emerging cluster structures generate "pseudo-labels" that guide feature selection, while the newly refined feature weights sharpen cluster boundaries in the next iteration.
The Virtuous Cycle of Joint Optimization
To understand why this unified approach works, we can look at how modern formulations implement this mutual feedback. A common strategy is to embed a clustering objective directly inside a sparse feature selection framework, allowing the model to leverage cluster structure dynamically.
This concept can be traced back to early Bayesian formulations. For instance, Shivakumar Vaithyanathan et al. introduced a unified Bayesian framework in Generalized Model Selection for Unsupervised Learning in High Dimensions that simultaneously determines the optimal number of clusters and the most relevant feature subset by maximizing marginal likelihood.
In more recent iterative frameworks, this co-optimization is often achieved by treating cluster assignments as latent variables or "pseudo-labels." Consider the wrapper approach proposed by Volker Roth et al. in Feature Selection in Clustering Problems. Instead of clustering raw data, the model reformulates clustering as a sparse regression problem within an Expectation-Maximization (EM) loop:
- The E-Step (Clustering): Compute fuzzy cluster assignments (pseudo-labels) based on the currently selected features.
- The M-Step (Feature Selection): Run a sparse regression (such as LASSO) on these pseudo-labels to update feature weights, effectively pruning variables that do not support the emerging cluster structure.
┌────────────────────────────────────────────────────────┐
│ │
│ Unsupervised Data │
│ │
└───────────────────────────┬────────────────────────────┘
│
▼
┌─────────────────────────────┐
│ Iterative Optimization │
│ Loop (e.g., EM Algorithm) │
└──────────────┬──────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Cluster Indicators │───────> Feature Selection │
│ (Pseudo-Labels) │ │ Weights ($l_{2,1}$) │
└───────────────────────┘ └───────────────────────┘
▲ │
└───────────────────────────────┘
By alternating between these two steps, the model ensures that the feature selection process is directly tailored to the classifier's or clusterer's decision-making process.
Leveraging Pseudo-Labels and Sparsity Constraints
In the absence of ground-truth labels, joint frameworks must generate their own internal supervision. A popular mechanism for doing this involves combining spectral clustering with sparsity-inducing norms.
For example, Zechao Li et al. developed the NDFS algorithm in Unsupervised Feature Selection Using Nonnegative Spectral Analysis. Here, spectral clustering and feature weight estimation are performed simultaneously. By enforcing a nonnegativity constraint on the cluster indicators, the model produces clean, interpretable pseudo-labels. These pseudo-labels act as a surrogate supervisor, allowing the model to apply supervised feature selection techniques—specifically utilizing an $l_{2,1}$-norm penalty—to enforce joint sparsity across features.
Similarly, Tetsuya Sakurai et al. proposed a joint framework in Spectral clustering and discriminant analysis for unsupervised feature selection that integrates spectral clustering with linear discriminant analysis. This joint optimization allows the evolving pseudo-labels to continuously refine the feature selection weights, which in turn improves the quality of the spectral embedding.
Preserving the Intrinsic Manifold
One risk of relying purely on pseudo-labels is that the model might converge to arbitrary or unstable partitions. To anchor the optimization, many joint frameworks incorporate manifold regularization, forcing the selected features to preserve the data's local geometric structure.
In Reconstruction-based Unsupervised Feature Selection: An Embedded Approach, Jundong Li et al. designed a framework (REFS) that selects features by minimizing data reconstruction error. Rather than using a fixed linear reconstruction, REFS automatically learns a data-dependent reconstruction function while simultaneously preserving local feature structures. This ensures that the chosen subset captures the most representative information without losing the underlying geometric layout of the data.
By combining global discriminative objectives with local manifold preservation, as done by Ronghua Shang et al. in Unsupervised feature selection based on kernel fisher discriminant analysis and regression learning, the models ensure that the selected features preserve both the geometric layout of the raw data and the separation between potential clusters.
Why It Works: Maximizing Class Margin Separability
The fundamental reason joint optimization outperforms sequential pipelines lies in the optimization landscape. When feature selection is decoupled from clustering, the selection criteria (such as maximum variance) might preserve features that represent global noise rather than cluster-specific signals.
By contrast, a joint framework actively shapes the feature space to maximize cluster separability. For example, the UFCM model introduced by Sen Wang et al. in Unsupervised Feature Analysis with Class Margin Optimization embeds K-means clustering directly into a maximum-margin framework. Because the feature weights are optimized to maximize the margin between the emerging clusters, the system actively suppresses features that blur cluster boundaries.
This synergy is highly effective in domain-specific applications where noise is prevalent. In single-cell RNA-seq analysis, Shuqing He et al. demonstrated in G3DC: a Gene-Graph-Guided selective Deep Clustering method for single cell RNA-seq data that performing feature selection and deep clustering within a unified objective ensures that the selected genes are precisely those that contribute most to differentiating distinct cell types, rather than merely those with high overall variance.
Beyond Global Selection: Personalized and Grouped Features
Unifying feature selection and clustering also unlocks flexible architectures that are mathematically impossible under sequential pipelines. Traditional feature selection assumes that a single global subset of features is optimal for the entire dataset. However, in complex datasets, different clusters might be defined by entirely different feature subsets.
To address this, Yen-Yu Lin et al. developed a framework in Clustering Complex Data with Group-Dependent Feature Selection that assigns a dedicated "expert" classifier to each cluster. The algorithm simultaneously partitions the data and selects the most relevant features for each specific group.
Taking this granularity a step further, Jundong Li et al. introduced UPFS in Unsupervised Personalized Feature Selection. By integrating feature selection with pseudo-label generation, UPFS simultaneously identifies a set of shared features and instance-specific features without requiring any ground-truth labels:
$$\min_{\mathbf{W}, \mathbf{P}} \mathcal{L}_{\text{clustering}}(\mathbf{X}, \mathbf{P}) + \alpha \|\mathbf{W}_{\text{shared}}\|_{2,1} + \beta \sum_{i=1}^n \|\mathbf{w}_i\|_1$$
Where $\mathbf{P}$ represents the cluster indicators (pseudo-labels), $\mathbf{W}_{\text{shared}}$ captures the global feature weights, and $\mathbf{w}_i$ allows for instance-specific (personalized) feature deviations. Such an architecture can only function because the cluster structures and the personalized feature weights are refined concurrently, allowing each task to provide the necessary context for the other.
Conclusion
By turning a sequential, two-step pipeline into a unified, iterative optimization problem, joint feature selection and clustering resolves the fundamental tension of high-dimensional unsupervised learning. It allows the data's latent structure to discover the features, and those features to refine the structure, leading to more robust, interpretable, and discriminative representations.
Key Foundational Work
Generalized Model Selection for Unsupervised Learning in High Dimensions (1999) — contributed 1 concept elements to this direction.
Iterative Construction of Sparse Polynomial Approximations (1991) — contributed 1 concept elements to this direction.