Feature Selection via Scaling Factor Integrated Multi-Class Support Vector Machines

Jinglin Xu, Feiping Nie, Junwei HanIJCAI 2017 · DOI

Introduction

In the era of "Big Data," researchers frequently encounter the "curse of dimensionality," particularly in fields like genomics where datasets contain thousands of features but limited samples. High dimensionality often leads to overfitting and increased computational costs. While feature selection is a common remedy, existing approaches have limitations: filter methods ignore feature redundancies, wrapper methods are computationally prohibitive, and standard embedded methods like $\ell_1$-SVM often lack flexibility for multi-class problems. This paper proposes a novel multi-class Support Vector Machine (SVM) framework that integrates a flexible "scaling factor" to automate feature selection by adaptively searching for optimal weight sparsity.

Methodology

The core innovation is the "Scaling Factor Integrated Multi-Class SVM," which introduces a scaling vector $\theta$ into the standard SVM objective.

  1. Scaling Factor Integration: Each feature $v$ is assigned a scaling factor $\theta_v$, allowing the model to learn the relative importance of features directly.
  2. Flexible Sparsity via $p$-power: A parameter $p \ge 1$ is introduced to create a flexible regularization term by raising the scaling factor to the power of $p/2$. This formulation bridges the gap between structured sparsity ($\ell_{2,1}$-norm when $p=1$) and the ideal but non-convex $\ell_{2,0}$-norm (as $p \to \infty$).
  3. Optimization Algorithm: The authors derive an iterative algorithm that alternates between updating scaling factors $\theta$ and updating the weight matrix $W$ using a modified Crammer and Singer multi-class SVM solver. Two variants are evaluated: SL21 ($p=1$) and SL2P ($p > 1$).

Experimental Setup

The framework was evaluated on six high-dimensional biological benchmark datasets: LUNG (3,312 features), CAR (9,182 features), BrainT2 (10,367 features), SRBCT (2,229 features), GLIOM (4,434 features), and MLLML (12,582 features). All datasets were standardized prior to analysis.

Results and Discussion

The proposed SL2P method consistently outperformed traditional filter methods (e.g., InfoGain, ChiSquare) and state-of-the-art SVM-based feature selection methods (CS and L21).

Classification accuracy of different methods across six benchmark datasets as a function of the number of selected features.

Accuracy and Sparsity

As shown in Figure 1, SL2P achieves superior classification accuracy across various feature subset sizes. Notably, on the MLLML dataset, SL2P reached 100% accuracy with only 40 features, outperforming competitors by a significant margin. The flexible $p$ parameter in SL2P proved crucial; for instance, on BrainT2 with 20 features, SL2P (~82% accuracy) vastly outperformed the fixed SL21 variant (~61% accuracy).

Convergence Analysis

The iterative optimization process demonstrates robust efficiency. Empirical results show that the objective function values for all datasets converge rapidly, typically within 10 to 50 iterations.

Convergence curves showing objective function values versus the number of iterations across five datasets.

Conclusion

The Scaling Factor Integrated Multi-Class SVM provides a mathematically elegant and practically effective solution for high-dimensional feature selection. By bridging the sparsity gap through a tunable $p$ parameter, the SL2P model adaptively identifies discriminative features across multiple classes. Future work may explore extending this framework to unsupervised learning and optimizing hyperparameter tuning for larger-scale applications.

Summary updated

BibTeX
@inproceedings{xu2017feature,
  author = {Jinglin Xu and Feiping Nie and Junwei Han},
  title = {{Feature Selection via Scaling Factor Integrated Multi-Class Support Vector Machines}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/442},
}
Figure