TL;DR
This paper investigates the trade-off between computational cost and performance in text classification by analyzing stacking ensembles of neural and non-neural models. They introduce a novel "oracle" method to predict optimal ensemble configurations, demonstrating that effective and efficient text classification can be achieved with significantly reduced computational resources.
What Problem Does This Paper Address?
Modern neural network models, particularly transformers like BERT and XLNet, have achieved state-of-the-art results in Automatic Text Classification (ATC). However, their impressive performance comes at a substantial computational cost, making them impractical for many applications. Ensemble methods, like stacking, offer a potential solution by combining simpler models to achieve high effectiveness, potentially with lower costs. This paper addresses the critical question of whether stacking can provide comparable or better performance than the best single model while significantly reducing computational expense, and proposes a method to predict these cost-effective ensembles.
Methods
The core of this work lies in a comprehensive experimental study of stacking ensembles. They combine various base classifiers, including powerful neural models (BERT, XLNet) and traditional non-neural methods (LinearSVM, kNN, Logistic Regression, XGBoost), with different text representations (TFIDF, FastText, PTE, MetaFeatures). The key novelty is the proposed "Oracle" method. This oracle uses a fraction of the training data to predict the best ensemble configuration for different scenarios:
- RQ1 (Cost-Constrained): Find an ensemble that is as effective as the best single model but significantly faster.
- RQ2 (Time-Constrained): Find an ensemble that is as effective as the best single model with a computational cost no greater than the best single model.
- RQ3 (Effectiveness-Focused): Find the best possible ensemble regardless of cost.
The oracle employs a greedy strategy, iteratively adding base algorithms that improve effectiveness while considering efficiency constraints.
Datasets
The study utilizes a range of datasets to evaluate their methods:
- Large-scale: AG's News (AGNews) and IMdB Reviews.
- Mid-sized: 20 Newsgroups (20NG), WebKB (WebKB), Reuters (Reut), and ACM Digital Library (ACM).
These datasets cover varying sizes, class distributions (balanced and imbalanced), and document lengths, providing a robust evaluation environment.
Key Insights and Ideas
- Stacking for Efficiency: Stacking ensembles can indeed achieve performance comparable to or better than the best single base model with significantly reduced computational costs, especially under time constraints.
- Meta-Features Shine: Recent "meta-features" representations consistently appear in the best-performing ensemble combinations, suggesting their effectiveness in capturing relevant information for text classification.
- Neural + Meta-Features Synergy: Combinations of powerful neural models (like BERT) with meta-features show particularly strong results, especially on larger datasets.
- Oracle's Predictive Power: The proposed oracle method can effectively predict cost-effective ensemble configurations using only a fraction of the training data, offering a practical solution for model selection.
- Dataset-Specific Solutions: The optimal stacking strategy is highly dependent on the dataset, emphasizing the need for adaptable approaches rather than a one-size-fits-all solution.
Key Results
- RQ1 (Cost-Constrained): In 4 out of 6 datasets, ensembles achieved statistically significant gains or ties in effectiveness with substantial speedups (up to 7.16x).
- RQ2 (Time-Constrained): In 5 out of 6 datasets, ensembles achieved effectiveness gains with no increase in computational cost compared to the best base model.
- RQ3 (Effectiveness-Focused): Removing time constraints led to further effectiveness gains, often by including more costly classifiers.
- Oracle Performance: The oracle successfully predicted effective ensembles, achieving wins or ties in 17 out of 36 scenarios (across RQs, datasets, and metrics), often with significantly reduced prediction costs.
Limitations and Future Work
- Oracle's Prediction Accuracy: While generally effective, the oracle's prediction accuracy can be impacted by the difficulty of predicting the best-performing model when using limited training data, particularly for complex neural models.
- IMdB Dataset Challenge: The IMdB dataset presented challenges for the oracle, especially when predicting the best neural model.
- Future Directions: The authors suggest exploring different oracle configurations, incorporating multi-objective feature selection, considering labeling effort as a constraint, and applying the oracle to other domains like recommender systems. They also highlight the potential of meta-features and BERT in future stacking research.
Summary updated