Abstract
This paper introduces a Bayesian framework for Automatic Relevance Determination (ARD) within Least Squares Support Vector Machine (LS-SVM) classifiers. By incorporating a weighting matrix into the kernel function and optimizing it across three hierarchical levels of inference, the proposed model can automatically identify and prune irrelevant input features. This process leads to improved generalization and reduced model complexity. Experimental validation on synthetic datasets shows that the ARD-based LS-SVM effectively isolates predictive features, achieving performance close to theoretical limits.
Introduction
In high-dimensional machine learning tasks, identifying relevant features is essential to prevent overfitting and ensure robust performance. While techniques like Automatic Relevance Determination (ARD) have been successfully integrated into Multi-Layer Perceptrons (MLPs), their application to Support Vector Machines (SVMs) is complicated by the "kernel trick," which often relies on an implicit mapping to feature space. This paper bridges this gap for Least Squares SVMs (LS-SVMs), a variant that simplifies the optimization problem into a set of linear equations.
Methodology
The authors propose a three-level Bayesian inference framework to handle model parameters, hyperparameters, and feature relevance:
- Level 1 (Model Parameters): The basic weights $w$ and bias $b$ are inferred by minimizing a cost function, corresponding to the maximum a posteriori (MAP) estimate.
- Level 2 (Hyperparameters): Regularization parameters ($\mu$ and $\zeta$) are optimized to balance model complexity against training error.
- Level 3 (Model Comparison & ARD): A diagonal weighting matrix $U$ is introduced into the kernel function (e.g., RBF). Each diagonal element $u(k)$ scales a specific input feature. By maximizing the marginal likelihood (evidence), the algorithm learns the importance of each feature.
If a feature is irrelevant, its corresponding weight $u(k)$ shrinks toward zero during the optimization process, allowing for systematic feature pruning.
Experimental Results
The approach was evaluated using a synthetic binary classification dataset consisting of 250 training and 1000 test points. The original two-dimensional problem was augmented with two "junk" features: a noisy version of the first input and pure Gaussian noise.
As shown in the figure above, the ARD algorithm successfully distinguished between relevant and irrelevant inputs. The weights for the two truly predictive features remained high, while the weights for the noisy features converged toward zero over successive iterations.
The impact on performance was significant:
- Initial Performance (4 inputs): 88.5% test accuracy.
- Optimized Performance (after ARD pruning): 91.0% test accuracy.
This result is remarkably close to the theoretical maximum performance of 92.0% for this dataset.
Conclusion
The Bayesian LS-SVM framework provides a principled, automated method for feature selection without the need for a separate validation set. While computationally more intensive than standard LS-SVMs due to the iterative inference process, it offers a clear path toward more parsimonious and accurate models. Future research may explore the use of full weighting matrices to capture feature correlations and the application of this framework to a broader range of kernel functions.
Summary updated