Abstract
Support Vector Machines (SVMs) are popular tools for data mining tasks such as classification, regression, and density estimation. However, original SVM (C-SVM) only considers local information of data points on or over the margin. Therefore, C-SVM loses robustness. To solve this problem, one approach is to translate (i.e., to move without rotation or change of shape) the hyperplane according to the distribution of the entire data. But existing work can only be applied for 1-D case. In this paper, we propose a simple and efficient method called General Scaled SVM (GS-SVM) to extend the existing approach to multi-dimensional case. Our method translates the hyperplane according to the distribution of data projected on the normal vector of the hyperplane. Compared with C-SVM, GS-SVM has better performance on several data sets.
Introduction
Standard Support Vector Machines (C-SVMs) are fundamentally "local" learners. Their decision boundaries are determined exclusively by support vectors—data points located on or near the margin. Consequently, the global distribution of the data is largely ignored. This focus can lead to sub-optimal performance, particularly when classes exhibit significantly different variances or spreads. While "Scaled SVM" was previously proposed to address this by shifting the boundary toward the class with smaller variance, its application was limited to one-dimensional scenarios. This paper introduces General Scaled SVM (GS-SVM), a method that generalizes this scaling logic to multi-dimensional datasets by incorporating global distribution information.
Methodology
GS-SVM improves robustness by incorporating global information through a three-step post-processing approach:
- Initial Optimization: A standard C-SVM is trained to identify the initial optimal hyperplane, defined by a weight vector $w$ and bias $b$.
- Dimensionality Reduction via Projection: All data points are projected onto the normal vector of the hyperplane. This reduces the multi-dimensional problem to a 1D space, allowing for the measurement of each class's "spread" relative to the boundary.
- Hyperplane Translation: The algorithm calculates the "scale" (range) of each class in the projected space. Based on the ratio of these scales, the hyperplane is translated by a distance $\Delta$ toward the class with smaller variance. This shift occurs without altering the hyperplane's orientation.
Notably, GS-SVM is compatible with kernel methods, enabling its application to non-linear classification tasks in high-dimensional feature spaces.
Experimental Evaluation
The performance of GS-SVM was evaluated using synthetic 2D toy data and eight benchmark datasets from the UCI Machine Learning Repository. On the synthetic data, GS-SVM demonstrated its ability to adjust the decision boundary more logically than C-SVM, resulting in an accuracy increase from 96.67% to 97.5%.
Across the UCI benchmarks, GS-SVM consistently matched or outperformed C-SVM using both linear and Gaussian kernels. For instance, on the "sonar" dataset, accuracy rose from 73.72% to 75.10%. The results indicate that GS-SVM often achieves peak performance at lower values of the penalty parameter $C$ compared to standard SVM, as it utilizes the translation parameter $\Delta$ to minimize structural risk.
Conclusion
GS-SVM offers a simple yet effective way to enhance SVM robustness by integrating global distribution data into the decision-making process. By shifting the hyperplane based on class variance, it achieves better generalization on unseen data with negligible computational overhead. Future work will focus on extending this framework to multi-class classification and providing a deeper theoretical analysis of its generalization capabilities.
Summary updated