TL;DR
This paper introduces i-IF-Learn, a novel iterative framework that simultaneously performs feature selection and clustering for high-dimensional, noisy data. Its key innovation is an adaptive feature selection statistic that intelligently balances unsupervised signals with potentially unreliable pseudo-labels from early clustering steps, leading to more robust and interpretable results.
What Problem Does This Paper Address?
High-dimensional datasets, common in fields like genomics and single-cell analysis, present a significant challenge for unsupervised learning. The sheer number of features often obscures the true underlying structure, with only a small subset of "influential features" actually driving the clusters. Traditional clustering methods struggle with this "curse of dimensionality," leading to poor performance. While feature selection can help, performing it independently from clustering can miss opportunities for joint refinement. Moreover, iterative clustering methods, while promising, are prone to error propagation from noisy early cluster assignments. This paper aims to address these issues by developing a framework that iteratively refines both feature selection and clustering, while carefully managing the reliability of intermediate results.
Methods
The core of i-IF-Learn is an iterative framework that alternates between two main steps:
- Feature Selection: This step uses a novel composite score statistic that adaptively combines two sources of information:
- Pseudo-label supervised statistic: This leverages the current (potentially noisy) cluster assignments.
-
Unsupervised statistic: This relies on inherent data properties. The framework dynamically adjusts the weight given to the supervised statistic based on its estimated reliability, mitigating error propagation. This is followed by a Higher Criticism Thresholding (HCT) procedure to select the influential features.
-
Clustering: After selecting a subset of influential features, the data is projected into a low-dimensional space using either Principal Component Analysis (PCA) or Laplacian Eigenmaps. The latter, which captures non-linear manifold structures, is found to be particularly effective. K-means clustering is then applied to these low-dimensional embeddings to generate updated cluster labels.
The process iterates until the selected feature set stabilizes or a maximum number of iterations is reached. The framework also includes an initialization step, often using the IFPCA method, to provide an initial set of labels and features.
Datasets
The authors evaluated their i-IF-Learn framework on a variety of real-world datasets:
- Gene Microarray Datasets: 10 datasets were used, characterized by high dimensionality and varying cluster separation.
- Single-cell RNA Sequencing (scRNA-seq) Datasets: 8 datasets were used, which are typically sparser and noisier than microarray data.
These datasets are publicly available, and the paper provides links for access. The authors also conducted extensive synthetic experiments to evaluate the method's performance under controlled linear and non-linear settings, and to analyze feature selection performance.
Key Insights and Ideas
- Adaptive Pseudo-label Supervision: The most significant contribution is the adaptive weighting of pseudo-label supervision. This allows the model to leverage early clustering results without being overly penalized by their potential inaccuracies, a common pitfall in iterative methods.
- Iterative Joint Refinement: The iterative nature of i-IF-Learn, where feature selection and clustering mutually inform each other, is crucial for uncovering weak signals and achieving better performance than static approaches.
- Laplacian Eigenmaps for Non-linear Structures: The paper highlights the effectiveness of Laplacian Eigenmaps for capturing complex, non-linear manifold structures in the data, leading to superior clustering results compared to linear methods like PCA.
- Interpretable Feature Subsets: Beyond just clustering, i-IF-Learn outputs a set of influential features, providing valuable insights into the underlying data structure and the drivers of the clusters. This interpretability is a key advantage, especially in domains like genomics.
- Downstream Enhancement: Using the selected influential features as a preprocessing step significantly boosts the performance of other state-of-the-art clustering and dimensionality reduction methods (e.g., DeepCluster, UMAP, VAE), underscoring the importance of targeted feature selection.
Key Results
- Superior Clustering Performance: i-IF-Learn consistently outperformed classical and deep clustering baselines across both gene microarray and scRNA-seq datasets, as measured by Accuracy and Adjusted Rand Index (ARI). For instance, it achieved the lowest average rank and regret on most datasets.
- Improved Downstream Models: When used as a preprocessing step, i-IF-Lap significantly improved the performance of UMAP, DeepCluster, and VAE, demonstrating the value of its feature selection.
- Robustness: The method showed robustness to different initialization schemes and parameter choices, indicating its reliability in practice.
- Effective Feature Selection: Synthetic experiments showed that i-IF-Learn significantly reduced False Positive Rates (FPR) and False Discovery Rates (FDR) compared to static methods like IFPCA, while achieving high True Positive Rates (TPR) and True Discovery Rates (TDR), especially as weak signals became stronger.
Limitations and Future Work
- Marginal Feature Screening: The current feature selection step evaluates features individually, potentially missing important pairwise or block interactions. Future work could explore incorporating methods that capture these joint effects.
- Global Feature Set: i-IF-Learn identifies a single global set of influential features. The authors suggest developing methods to detect distinct, cluster-specific feature subsets for more granular insights.
- Theoretical Guarantees: While theoretical guarantees are provided for feature selection and label recovery under certain conditions, practical performance can still be sensitive to parameter choices and data characteristics.
- Computational Cost: The iterative nature of the algorithm can lead to a higher computational cost compared to single-pass methods, although the authors state their implementation is relatively lightweight.
Summary updated