Learning Using Unselected Features (LUFe)

Joseph G. Taylor, Viktoriia Sharmanska, Kristian Kersting, David Weir, Novi Quadrianto

Introduction

In machine learning, feature selection is a critical yet paradoxical process. Standard methods improve efficiency by discarding "less informative" features to create smaller, faster models. However, this often results in the loss of valuable contextual information. Learning Using Unselected Features (LUFe) proposes a "waste not, want not" approach: it leverages these discarded features during the training phase to better constrain the decision boundary. The result is a more accurate model that maintains a small footprint and high speed during deployment.

The Problem: Local vs. Global Views

Feature selection typically falls into two categories:

  1. Combinatorial methods (filters and wrappers) select a small subset of features, providing a "local view" that is efficient at test time but loses information from discarded features.
  2. Continuous/Embedded methods maintain a "global view" by considering all features, which improves accuracy but increases computational complexity and model size.

LUFe bridges this gap, aiming to capture the accuracy benefits of a global view during training without the associated costs during deployment.

Methodology: The LUFe Framework

Inspired by "Learning Using Privileged Information" (LUPI), LUFe assigns unselected features a secondary, guiding role during training. In a standard Support Vector Machine (SVM), the objective is to minimize classification error on selected features. LUFe modifies this by using unselected features to define a data-dependent upper bound on the loss function.

Essentially, the unselected features act as a "teacher." If these features indicate that a specific data point is noisy or difficult to classify, the model adjusts its treatment of that point when optimizing the primary features.

  • At Train Time: The optimization considers both selected and unselected features.
  • At Test Time: The unselected features are discarded. The final classifier only requires the small subset of selected features, resulting in zero additional computational cost during deployment.

Experimental Setup

The researchers evaluated LUFe using the TechTC-300 collection, comprising 49 textual datasets of web pages categorized by topic. These datasets were pre-processed to remove short words, and baseline SVM error rates ranged from 0.0 to 0.4, providing a robust testing environment.

Key Results

Empirical results across the 49 datasets demonstrated significant performance gains. For a subset of 300 features ($K=300$), LUFe-RFE outperformed standard Recursive Feature Elimination (RFE) in 89.8% of cases, with a mean improvement of 1.97%. Notably, LUFe-RFE often outperformed standard SVMs using all features (83.67% of cases), despite using a fraction of the features at test time.

Comparison of LUFe-RFE performance against standard RFE and the full feature set (ALL) for K=300 and K=500.

These improvements were consistent across different feature counts ($K=300$ and $K=500$) and selection metrics, including ANOVA F-scores, where LUFe improved mean accuracy by 2.65% over the standard filter method.

Key Insights: The Value of "Discarded" Data

A core finding of the research is that features which are not "the best" for a decision boundary still contain vital information about data structure. By using these as constraints rather than direct inputs, the model avoids overfitting while benefiting from the extra information.

Mean improvement by LUFe relative to the percentage of top unselected features used during training.

Furthermore, the researchers identified a "sweet spot" for feature utilization. As shown in the figure above, using the top 60-80% of discarded features often yielded the best results, suggesting that while unselected features are useful, the very lowest-ranked features may still introduce noise.

Limitations and Future Work

While LUFe offers zero-cost deployment benefits, it does introduce training overhead due to the more complex optimization problem (similar to SVM+). Currently, the framework is optimized for SVMs; adapting the mathematical formulation for other classifiers like Neural Networks remains a future challenge. Additionally, the authors suggest that joint optimization—simultaneously deciding which features to select and how to use the unselected ones—could further enhance the framework's effectiveness.

Summary updated

Abstract

Feature selection has been studied in machine learning and data mining for many years, and is a valuable way to improve classification accuracy while reducing model complexity. Two main classes of feature selection methods - filter and wrapper - discard those features which are not selected, and do not consider them in the predictive model. We propose that these unselected features may instead be used as an additional source of information at train time. We describe a strategy called Learning using Unselected Features (LUFe) that allows selected and unselected features to serve different functions in classification. In this framework, selected features are used directly to set the decision boundary, and unselected features are utilised in a secondary role, with no additional cost at test time. Our empirical results on 49 textual datasets show that LUFe can improve classification performance in comparison with standard wrapper and filter feature selection.
BibTeX
@inproceedings{taylor2016learning,
  author = {Joseph G. Taylor and Viktoriia Sharmanska and Kristian Kersting and David Weir and Novi Quadrianto},
  title = {{Learning Using Unselected Features (LUFe)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure