In supervised learning, labels act as a compass, explicitly telling a model which features correlate with the target. In unsupervised learning, we are effectively flying blind. Without ground-truth labels, how can an algorithm distinguish between a feature that defines the underlying structure of the data and one that is merely noise?
The core intuition behind unsupervised feature selection is that informative features are those that preserve the intrinsic structure of the data. Whether that structure is defined by local geometry, cluster stability, or consistency across multiple views, the goal is to identify a subset of variables that allows the data to "speak for itself."
Preserving the Data Manifold
One of the most powerful ways to identify relevance without labels is to look at the data's local geometry. If two data points are close together in the original high-dimensional space, a "good" feature should help maintain that proximity.
He et al. pioneered this approach with the Laplacian Score. By modeling the dataset as a nearest-neighbor graph, the authors demonstrated in Laplacian Score for Feature Selection that features can be ranked based on their ability to preserve this graph's structure. Similarly, Yu et al. proposed SNFS (Stochastic Neighbor-preserving Feature Selection) in Unsupervised Feature Selection by Preserving Stochastic Neighbors, which selects features that not only keep similar points together but actively push dissimilar ones apart, mimicking the way human perception organizes local neighborhoods.
Sparsity and Self-Representation
Another perspective treats feature selection as a compression problem: which features are sufficient to reconstruct the rest of the data? This often involves using sparsity-inducing penalties like the LASSO to "unplug" redundant dimensions.
The authors of Feature Selection in Clustering Problems (Roth et al.) framed clustering as a sparse regression problem, identifying the most stable partitions by ignoring noise features. Modern deep learning approaches have evolved this into AutoEncoder Feature Selection (AEFS). As shown in AutoEncoder Inspired Unsupervised Feature Selection, neural networks can learn non-linear relationships and use Group Lasso penalties to identify a "basis" of the feature space. Sokar et al. further optimized this in Quick and robust feature selection by using sparse training to rank features based on "neuron strength," achieving high accuracy with significantly less energy and memory.
Information Bottlenecks and Multi-View Consistency
When data is available from multiple sources or "views" (e.g., an image and its sketch), relevance can be defined by what is consistent across those views. The intuition is that the "signal" is shared, while the "noise" is view-specific.
Dutta et al. introduced the Multi-View Information Bottleneck (MIB) in Learning Robust Representations via Multi-View Information Bottleneck, which discards information unique to a single view to retain only core, task-relevant features. Building on this, Wang et al. proposed MVEB in Self-Supervised Learning with Multi-View Entropy Bottleneck, which balances view alignment with entropy maximization to learn "minimal sufficient representations" without any human-provided labels.
Beyond Global Selection
Recent work suggests that a single "best" subset of features might not exist for the entire dataset. Poon et al. argue in Variable Selection in Model-Based Clustering that high-dimensional data is multifaceted, meaning it can be clustered in multiple valid ways depending on which features one prioritizes. Taking this further, Li et al. introduced Personalized Feature Selection in Unsupervised Personalized Feature Selection, which identifies both shared features and instance-specific features, recognizing that what is relevant for one data point might be noise for another.
Key Foundational Work
Input pruning for neural gas architectures (2001) — contributed 2 concept elements to this direction.