Abstract
Supervised text classification is often hindered by the high cost of manual data labeling. This paper presents WTC-WBR, a novel framework for weakly-supervised text classification that utilizes only a small set of representative category words. By integrating a weakly self-training mechanism with Wasserstein Barycenter Regularization (WBR), the model effectively captures the geometric structure of text features in high-dimensional space. Experimental results on benchmark datasets demonstrate that WTC-WBR significantly outperforms existing weakly-supervised methods and approaches the performance of semi-supervised and fully supervised models.
Introduction
While deep learning has revolutionized text classification, the "labeling bottleneck" remains a significant hurdle. Supervised models typically require thousands of annotated documents to achieve high accuracy. Weakly-supervised learning offers a more efficient alternative by using "seed words" (e.g., "sports," "politics") to guide classification. However, these signals are often sparse and noisy; for instance, in datasets like AG News, only a small fraction of documents actually contain the provided category words. Previous attempts to address this via pseudo-labeling or word expansion often failed to account for the underlying geometric distribution of data, leading to suboptimal performance.
Methodology
The proposed WTC-WBR framework addresses these challenges through a three-stage process designed to refine noisy initial signals into robust classification boundaries.
1. Pseudo-label Initialization
The process begins by identifying the presence of user-provided category words within the unlabeled corpus. Documents containing specific seed words are assigned initial pseudo-label scores, providing a starting point for the model to learn class-specific features.
2. Weakly Self-Training
To mitigate the noise inherent in initial pseudo-labels, WTC-WBR employs an iterative self-training loop. The model combines these initial labels with its own "sharpened" predictions. An annealing schedule is used to gradually shift the model's reliance from the sparse seed words to its own high-confidence learned patterns as training progresses.
3. Wasserstein Barycenter Regularization (WBR)
The core innovation of this work is the application of Wasserstein Barycenter Regularization. The model treats each class as a trainable "barycenter" (geometric center) within the deep feature space. Unlike standard Euclidean distance, the Wasserstein distance (Earth Mover's Distance) accounts for the optimal transport between distributions. This allows the model to capture the complex, non-linear geometry of language representations provided by pre-trained models like BERT.
By forcing text features to cluster around these class prototypes using the Sinkhorn algorithm for computational efficiency, the model ensures that semantically similar documents are grouped together even if they lack the original seed words.
Experimental Setup and Results
The model was evaluated on three standard benchmarks: IMDB (sentiment analysis), AG News (topic classification), and DBPedia (ontology classification). In all experiments, supervision was limited strictly to the label names themselves.
Performance Comparison
WTC-WBR consistently outperformed state-of-the-art weakly-supervised baselines such as LOTClass and X-Class:
- AG News: The model achieved 89.7% accuracy, surpassing previous weakly-supervised benchmarks and coming within 2% of semi-supervised models trained with 2,500 labels per class.
- DBPedia: The model reached an impressive 98.4% accuracy, nearly matching the 99.3% accuracy of a fully supervised BERT model.
Geometric Analysis
The effectiveness of WBR is most evident in the feature space distribution. t-SNE visualizations reveal that WTC-WBR produces significantly tighter and more distinct clusters compared to standard self-training methods. This geometric alignment explains the model's superior discriminative power and its ability to generalize from sparse initial signals.
Conclusion
WTC-WBR demonstrates that explicitly modeling the geometric structure of the feature space can overcome the limitations of sparse supervision in text classification. By leveraging Wasserstein barycenters, the model achieves near-supervised performance with minimal human input. Future work may involve exploring automated category word expansion and applying these geometric principles to multi-label and cross-lingual tasks.
Summary updated