When dealing with high-dimensional data, traditional machine learning algorithms frequently collide with the "curse of dimensionality." In a space with hundreds or thousands of features, data points become roughly equidistant from one another, causing standard distance metrics like Euclidean distance to lose their meaning.
The core insight of the local feature relevance paradigm is that global feature selection is an artificial constraint. In complex, real-world datasets, features are rarely uniformly important across the entire data space. Instead, distinct groups of data points cluster within specific, low-dimensional subspaces of the full feature set. By dynamically identifying which features matter for which specific sub-regions or clusters, models can filter out overwhelming noise and uncover patterns that are completely invisible from a global perspective.
The Failure of Global Metrics
To understand why local feature relevance is necessary, consider a dataset where points are distributed across 100 dimensions. If a meaningful cluster forms based on tight alignment in just 5 of those dimensions, a global distance metric will still compute proximity across all 100 dimensions. The variance and noise from the 95 irrelevant dimensions will completely overwhelm the signal from the 5 relevant ones.
As Jianyu Miao et al. note in A Comprehensive Survey on Subspace Clustering: Methods and Applications, high-dimensional data points typically exhibit similarity only within these localized, low-dimensional slices. If we attempt to apply global feature selection—such as dropping dimensions that appear noisy across the entire dataset—we risk discarding variables that are critically informative for specific sub-populations. For instance, in patient health profiling, Yicheng He et al. demonstrate in Drug Selection via Joint Push and Learning to Rank that global feature selection routinely misses key variables that are only significant within specific, heterogeneous patient subgroups.
Adaptive Metrics and Projected Clustering
How do we find these hidden subspaces when we do not know the clusters or the relevant features in advance? The solution is to co-optimize the data partitioning and the feature weights iteratively.
Global Distance (Fails):
Point A •=======(Noise from 95 irrelevant dimensions)=======• Point B
Locally Adaptive Distance (Succeeds):
Point A •---(Projected onto 5 relevant dimensions)---• Point B
A foundational approach to this problem is Locally Adaptive Clustering (LAC), detailed by Dimitris Papadopoulos et al. in Clustering gene expression data in SQL using locally adaptive metrics. Instead of treating all dimensions equally, the algorithm assigns a local weight vector to each cluster.
The mechanism works by analyzing local variance:
- Start with an initial cluster assignment.
- Measure the variance of the data points along each dimension within that specific cluster.
- Dimensions with low variance indicate that the cluster points are tightly packed along that axis, meaning the feature is highly relevant to that cluster's definition. The algorithm assigns a high weight to this dimension.
- Dimensions with high variance indicate noise or irrelevance for this group, so they receive a low weight.
This localized weighting strategy transforms the distance metric for each cluster individually. In streaming data contexts where patterns shift over time, frameworks like HPStream—introduced by Jiawei Han 0001 et al. in A Framework for Projected Clustering of High Dimensional Data Streams—and HDDSTREAM, developed by Irene Ntoutsi et al. in Density-based Projected Clustering over High Dimensional Data Streams, dynamically update these preferred projections. By restricting distance calculations to these localized subspaces, they prevent high-dimensional noise from corrupting cluster boundaries.
Local Variable Selection in Regression and Trees
The power of local feature relevance extends beyond unsupervised clustering into supervised regression and classification. Just as clusters live in subspaces, complex functions often rely on different predictors depending on the region of the input domain.
Jerome H. Friedman pioneered this concept in Adaptive Spline Networks through Multivariate Adaptive Regression Splines (MARS). Rather than forcing a variable to influence the model globally, the architecture allows the model to ignore specific inputs within localized sub-regions of the data space.
Modern partition-wise frameworks formalize this behavior mathematically. As shown by Rex C. Y. Cheung et al. in Consistent Estimation for Partition-Wise Regression and Classification Models, partitioning the input domain allows different subsets of predictors to be active in different regions. This captures localized interactions that standard global models or shallow decision trees over-generalize.
This spatial heterogeneity is particularly evident in geographic data modeling. In Bayesian Cluster Geographically Weighted Regression for Spatial Heterogeneous Data, Wala Draidi Areed et al. demonstrate that a covariate can be a critical predictor in one geographic neighborhood while acting as pure noise in another. By calculating feature importance independently for localized models—a technique also leveraged by Ryan Zhenqi Zhou et al. in PyGRF: An improved Python Geographical Random Forest model and case studies in public health and natural disasters—the framework uncovers granular, spatially varying relationships that global averages obscure.
Learning Subspaces Probabilistically
While variance-thresholding and hard partitioning work well, they can struggle with overlapping clusters or soft boundaries. Nonparametric Bayesian methods handle this by treating local feature relevance as a probabilistic latent variable problem.
In Online Learning of a Dirichlet Process Mixture of Generalized Dirichlet Distributions for Simultaneous Clustering and Localized Feature Selection, Nizar Bouguila et al. introduce a framework that assigns a binary latent variable to every feature within every cluster.
Instead of forcing a hard projection, the model estimates the probability that a feature is "active" or "inactive" for a given cluster:
$$\mathbb{P}(\text{Feature } j \text{ is relevant} \mid \text{Cluster } k)$$
This allows the mixture model to learn a distinct feature relevance map for each group on the fly. If a feature is deemed irrelevant for cluster $A$, its distribution defaults to a global background noise distribution; if it is relevant for cluster $B$, the model leverages its cluster-specific parameters to tightly fit the data. This dual mechanism ensures that features considered noise for one group can still serve as vital signals for another, resulting in highly precise data partitioning without requiring prior knowledge of the underlying subspaces.
Key Foundational Work
Adaptive Spline Networks (1990) — contributed 2 concept elements to this direction.