Abstract
A method for multi-class and multi-label automated text categorization based on twin-SVM with naive Bayes ensemble is proposed. Twin-SVM classifiers give a solution to the multi-label problem. For multiclass situation, naive Bayes classifier constrains the belonging scope of a testing sample within a few most likely classes and greatly reduces the number of binary classifiers needed to make the final prediction. The benefits of the ensemble method are described and preliminary results with Reuters-21578 data set are also presented.
Introduction
Automated text categorization faces a persistent trade-off between computational efficiency and classification accuracy, particularly in multi-class and multi-label environments. Standard Support Vector Machines (SVMs), while precise, are inherently binary. Extending them to multi-class scenarios via "one-vs-one" or "one-vs-all" strategies leads to a computational bottleneck as the number of categories grows. Furthermore, these traditional strategies often struggle with multi-labeling—where a single document belongs to multiple categories—because they typically rely on a single-winner voting mechanism.
Methodology
The proposed ensemble architecture utilizes a two-stage process to balance speed and precision.
Stage 1: Naïve Bayes Coarse Filtering
The system first employs a Naïve Bayes (NB) classifier to perform "coarse filtering." NB is computationally inexpensive and serves to rank possible labels. By selecting only the top-ranked labels whose cumulative probability exceeds a threshold ($\theta$), the system significantly narrows the search space for the subsequent, more expensive SVM phase. This "sparsity" approach allows the system to focus on an average of 2.8 classes per sample instead of the full set of 10.
Stage 2: Twin-SVM Fine-tuning
For the labels identified by the NB filter, a "Twin-SVM" approach is used. Unlike standard binary SVMs that force a choice between two classes, this setup trains two separate classifiers for any pair of classes ($c_1, c_2$):
- SVM1: Distinguishes $c_1$ from samples belonging exclusively to $c_2$.
- SVM2: Distinguishes $c_2$ from samples belonging exclusively to $c_1$.
This dual-classifier structure allows a sample to be assigned to both classes if both SVMs return a positive signal, effectively solving the multi-label hurdle.
Experimental Results
The method was evaluated using the Reuters-21578 corpus (ModApte split), focusing on the top 10 most populous classes.
Performance Analysis
The ensemble method (Twin-SVM + NB) demonstrated superior performance across all metrics. In terms of speed, the proposed ensemble was more than twice as fast as the standard one-vs-one SVM approach during the testing phase.
As shown in the results table, the ensemble method achieved a full-set error rate of 18.39%, outperforming both the standard one-vs-one SVM (27.81%) and the standalone Naïve Bayes (34.8%). This improvement was consistent across every individual category tested, including acq, corn, crude, and earn.
Conclusion
The hybrid ensemble approach successfully combines the efficiency of generative models (Naïve Bayes) with the discriminative power of SVMs. While the current study focused on a limited set of categories, the results suggest that the "coarse-to-fine" strategy is highly effective for multi-label text categorization. Future work should explore the scalability of the Twin-SVM pairings for larger label sets and the automated tuning of the probability threshold $\theta$.
Summary updated