Stable Feature Selection with Support Vector Machines

Iman Kamkar, Sunil Kumar Gupta, Dinh Q. Phung, Svetha Venkatesh

Abstract

The support vector machine (SVM) is a popular method for classification, well known for finding the maximum-margin hyperplane. Combining SVM with $$l_{1}$$ -norm penalty further enables it to simultaneously perform feature selection and margin maximization within a single framework. However, $$l_{1}$$ -norm SVM shows instability in selecting features in presence of correlated features. We propose a new method to increase the stability of $$l_{1}$$ -norm SVM by encouraging similarities between feature weights based on feature correlations, which is captured via a feature covariance matrix. Our proposed method can capture both positive and negative correlations between features. We formulate the model as a convex optimization problem and propose a solution based on alternating minimization. Using both synthetic and real-world datasets, we show that our model achieves better stability and classification accuracy compared to several state-of-the-art regularized classification methods.
BibTeX
@inproceedings{kamkar2015stable,
  author = {Iman Kamkar and Sunil Kumar Gupta and Dinh Q. Phung and Svetha Venkatesh},
  title = {{Stable Feature Selection with Support Vector Machines}},
  booktitle = {Australasian Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1007/978-3-319-26350-2\_26},
}

Loading related content...

Figure