When training a machine learning model on high-dimensional data, we often face a fundamental challenge: many input features are irrelevant, noisy, or redundant. Instead of relying on separate statistical filters before training, a highly effective strategy is to let the model itself discover which features matter during optimization.
By treating the learned weights of a model as a direct proxy for feature importance, we can prune or mask low-weight features. This approach unifies feature selection and model optimization into a single pipeline, reducing dimensionality, accelerating inference, and preventing overfitting.
The Geometry of Weight-Based Selection
In a linear model or a Support Vector Machine (SVM), the learned weight vector directly dictates how much each feature influences the final prediction. Geometrically, the weight vector is orthogonal to the decision boundary. Features with large absolute weights drive the orientation of this boundary, while features with weights near zero contribute almost nothing to separating the classes.
As shown by Fan et al. in Fast Recognition of Multi-View Faces with Feature Selection, the criteria used to establish optimal separating boundaries in SVMs inherently rank feature importance. The features that contribute most to maximizing the margin between different classes receive the largest weights.
This geometric insight is leveraged across diverse domains. For instance, Tong et al. used SVM hyperplane weights to isolate highly discriminative phonotactic features in Selecting phonotactic features for language recognition, focusing resources on features that actively distinguish classes rather than those that merely occur frequently. Similarly, Okubo et al. utilized linear SVM weights to identify key behavioral predictors of student success in Learning Activity Features of High Performance Students.
Iterative Elimination vs. Direct Masking
Once we establish that weight magnitudes reflect feature importance, we must decide how to remove the uninformative ones. Two primary paradigms exist: iterative backward elimination and single-shot masking.
Iterative Elimination
In the iterative approach, a model is trained on the full feature set, a fraction of the lowest-weighted features are discarded, and the model is retrained on the remaining subset. This greedy search strategy—often called Recursive Feature Elimination (RFE)—prevents the model from losing performance when features have complex joint dependencies.
For example, Bahamonde et al. demonstrated in Feature subset selection for learning preferences: a case study that iteratively removing attributes with the lowest weights yields highly optimized, compact feature subsets.
More recently, Uribarri et al. applied this to time-series classification in Detach-ROCKET: sequential feature selection for time series classification with random convolutional kernels, stripping away up to 99% of randomly generated kernel features by repeatedly discarding those with the smallest absolute weights. Campbell et al. also utilized an alternating filter-wrapper approach based on SVM weight rankings to isolate key acoustic markers in Discriminative Keyword Selection Using Support Vector Machines.
Direct Masking
While iterative elimination is highly effective, retraining a model repeatedly over massive feature spaces is computationally expensive. A surprising and powerful counter-intuition is that retraining is often completely unnecessary.
In Avoidance of Model Re-Induction in SVM-Based Feature Selection for Text Categorization, Kolcz et al. demonstrated that simply "masking" (zeroing out) the weights of low-ranked features from a single pre-trained model yields classification performance nearly identical to full model retraining. Because the optimal decision boundary in high-dimensional spaces is remarkably robust to the removal of low-influence dimensions, we can prune features post-hoc without expensive optimization loops.
Scaling to Non-Linear and Structural Spaces
The intuition of weight-based pruning extends naturally to non-linear spaces via kernel expansions and neural network architectures.
Kernel Expansions
When using non-linear kernels (such as polynomial kernels), the feature space expands exponentially to capture feature combinations. Yet, the vast majority of these combinations have negligible weights.
In Fast Methods for Kernel-Based Text Analysis, Kudo et al. showed that explicitly expanding and thresholding these kernel weights allows models to discard the vast majority of the feature space, speeding up inference by orders of magnitude without sacrificing accuracy. Isozaki et al. similarly leveraged an expanded kernel framework in Efficient Support Vector Classifiers for Named Entity Recognition to identify and remove non-discriminative features, reducing memory overhead while mitigating overfitting.
Neural Networks
In neural networks, weight pruning can be applied structurally at the input layer. Rather than pruning individual, isolated connections throughout the network, entire input units (neurons) are removed based on their collective weight profiles or second-order derivatives.
As detailed by Stahlberger et al. in Fast Network Pruning and Feature Extraction by using the Unit-OBS Algorithm, pruning entire input units effectively performs automated feature selection, simplifying the model's input space and clarifying the underlying data relationships.
This philosophy of learning the architecture alongside the weights dates back to early connectionist models. Kassebaum et al. demonstrated in The Cocktail Party Problem: Speech/Data Signal Separation Comparison between Backpropagation and SONN that automated architecture construction eliminates redundant data processing by utilizing only the most relevant input features. Similarly, Isu et al. used weight-zeroing procedures during learning in Development of NEural network simulator for structure-activity COrrelation of molecules: Neco. to unmask the specific molecular properties driving predictions.
Why It Works: Contextual Evaluation
Weight-based feature pruning works because it aligns the feature selection criteria directly with the model's objective function. Traditional filter methods (like mutual information or chi-squared tests) evaluate features independently of the classifier and independently of each other. In contrast, weight-based pruning evaluates features in the context of all other features through the lens of the model's optimization landscape.
By discarding low-weight features, the model achieves two critical advantages:
- Regularization: Removing low-weight dimensions eliminates background noise, which prevents overfitting and can even boost classification accuracy. This phenomenon was observed by Attabi et al. in Weighted ordered classes - nearest neighbors: a new framework for automatic emotion recognition from speech, where weight-derived feature selection refined classification patterns by discarding uninformative relationships.
- Computational Efficiency: Shrinking the input space drastically reduces memory footprints and speeds up inference, making complex models viable for real-time applications.
Key Foundational Work
Development of NEural network simulator for structure-activity COrrelation of molecules: Neco. (1994) — contributed 1 concept elements to this direction.
The Cocktail Party Problem: Speech/Data Signal Separation Comparison between Backpropagation and SONN (1989) — contributed 1 concept elements to this direction.