Joint feature re-extraction and classification using an iterative semi-supervised support vector machine algorithm

Yuanqing Li, Cuntai Guan

Abstract: This paper addresses the challenge of feature extraction in scenarios with limited labeled data. The authors propose an iterative, semi-supervised algorithm that jointly optimizes feature extraction and classification. By using initial SVM predictions on unlabeled data to refine feature extraction through a feedback loop, the method significantly improves classification accuracy. The framework unifies Common Spatial Pattern (CSP) and Fisher Discriminant (FD) features by maximizing the Rayleigh coefficient. Experimental results on synthetic, UCI benchmark, and EEG datasets demonstrate the effectiveness and computational efficiency of the proposed approach compared to standard and transductive SVMs.

Introduction

A persistent challenge in pattern recognition is the "chicken-and-egg" problem between feature extraction and classification: high-quality features are necessary for accurate classification, but effective feature extraction typically requires a large labeled dataset. In fields like Brain-Computer Interfacing (BCI), labeled data is often scarce and expensive to obtain. Standard workflows treat these as separate, sequential steps, which can lead to "unreliable" features when training data is limited. This paper proposes a solution by integrating these steps into a joint, iterative process.

Methodology

The authors introduce an iterative algorithm that treats feature extraction and classification as a unified feedback loop. The process begins by training a standard SVM on a small labeled dataset. This model then predicts labels for a larger unlabeled dataset. These predicted labels, combined with the original labels, are used to "re-extract" features. This cycle repeats until the predicted labels stabilize.

A key contribution is the unification of three feature types—Common Spatial Pattern (CSP) and two variants of Fisher Discriminant features (FD1 and FD2)—under the mathematical framework of maximizing the Rayleigh coefficient. This coefficient measures the ratio of between-class variance to within-class variance. To ensure stability with small datasets, a regularization term ($\alpha I$) is added to the scatter matrices.

Comparison of CSP, FD1, and FD2 feature extraction methods within the unified Rayleigh coefficient framework.

The authors also propose a semi-supervised model selection technique. Instead of standard cross-validation, which is unreliable for tiny datasets, hyperparameters are chosen to maximize the Rayleigh coefficient across the entire combined (labeled and predicted) dataset.

Experimental Evaluation

The algorithm was validated using three types of data:

  1. Synthetic Data: 16-dimensional Gaussian distributions used to demonstrate convergence and the validity of Rayleigh-based model selection.
  2. UCI Benchmarks: Five real-world datasets (Cancer, Ionosphere, Diabetes, Dimdata, and German Credit) commonly used to test classification algorithms.
  3. BCI Competition 2005: EEG motor imagery data (Data set IVa), involving subjects performing mental tasks like imagining right-hand or foot movements.

Results and Discussion

The proposed method demonstrated significant performance gains. For instance, on the "Dim" dataset, accuracy improved from 81% (standard SVM) to 96.1%. It outperformed Transductive SVMs (TSVMs) in most UCI benchmarks while being significantly more computationally efficient; one test showed a reduction in processing time from 140 hours for a TSVM to just 48 seconds for the proposed algorithm.

The study highlights that feature spaces should be dynamic rather than static, evolving as the model learns the distribution of unlabeled data. The Rayleigh coefficient serves as an effective proxy for model selection, indicating that higher separability in the feature space often correlates with better classification performance.

Conclusion

The iterative semi-supervised approach effectively addresses the limitations of small labeled datasets in feature extraction. While the risk of "self-reinforcing errors" exists if the initial classifier is poor, the method shows great promise for real-time applications like BCIs. Future research will explore kernel-based nonlinear feature extraction and extensions to multi-class classification problems.

Summary updated

Abstract

The focus of this paper is on joint feature re-extraction and classification in cases when the training data set is small. An iterative semi-supervised support vector machine (SVM) algorithm is proposed, where each iteration consists both feature re-extraction and classification, and the feature re-extraction is based on the classification results from the previous iteration. Feature extraction is first discussed in the framework of Rayleigh coefficient maximization. The effectiveness of common spatial pattern (CSP) feature, which is commonly used in Electroencephalogram (EEG) data analysis and EEG-based brain computer interfaces (BCIs), can be explained by Rayleigh coefficient maximization. Two other features are also defined using the Rayleigh coefficient. These features are effective for discriminating two classes with different means or different variances. If we extract features based on Rayleigh coefficient maximization, a large training data set with labels is required in general; otherwise, the extracted features are not reliable. Thus we present an iterative semi-supervised SVM algorithm embedded with feature re-extraction. This iterative algorithm can be used to extract these three features reliably and perform classification simultaneously in cases where the training data set is small. Each iteration is composed of two main steps: (i) the training data set is updated/augmented using unlabeled test data with their predicted labels; features are re-extracted based on the augmented training data set. (ii) The re-extracted features are classified by a standard SVM. Regarding parameter setting and model selection of our algorithm, we also propose a semi-supervised learning-based method using the Rayleigh coefficient, in which both training data and test data are used. This method is suitable when cross-validation model selection may not work for small training data set. Finally, the results of data analysis are presented to demonstrate the validity of our approach.
BibTeX
@article{li2008joint,
  author = {Yuanqing Li and Cuntai Guan},
  title = {{Joint feature re-extraction and classification using an iterative semi-supervised support vector machine algorithm}},
  journal = {Machine Learning},
  year = {2008},
  doi = {10.1007/s10994-007-5039-1},
}
Figure