Handling Imbalanced Dataset in Multi-label Text Categorization using Bagging and Adaptive Boosting

Genta Indra Winata, Masayu Leylia KhodraarXiv 2018 · arXiv

Abstract

Multi-label text classification often suffers from severe class imbalance, where models prioritize majority classes at the expense of rare but critical categories. This study investigates the application of ensemble techniques—specifically Bagging and Adaptive Boosting (AdaBoost.MH)—to address this issue within the context of Indonesian government complaint data. By evaluating these methods against Binary Relevance and Label Powerset baselines across various base classifiers, the research demonstrates that ensemble strategies can significantly improve performance on minority labels. Results indicate that while AdaBoost.MH is effective for minimizing overall error (Hamming Loss), Bagging provides superior improvements in subset accuracy and minority class recognition without the overfitting risks associated with traditional data-level resampling.

Introduction

In real-world multi-label classification tasks, such as categorizing citizen complaints, data distribution is rarely uniform. A city government might receive thousands of reports regarding infrastructure like "potholes" but only a handful concerning niche administrative errors. Standard machine learning classifiers are often "lazy," optimizing for majority classes to achieve high overall accuracy while effectively ignoring minority labels.

In a multi-label context—where a single instance may belong to multiple categories simultaneously—this imbalance becomes increasingly complex. This paper explores "algorithm-level" solutions, modifying the learning process through ensemble methods to improve performance on rare classes without causing the model to overfit, a common side effect of "data-level" fixes like oversampling.

Methodology

The researchers utilized the LAPOR dataset, comprising 5,151 Indonesian language complaints categorized into 70 different labels. The dataset exhibits significant imbalance, with 55 of the 70 classes identified as minority categories. Preprocessing involved formalization and stemming for the Indonesian language, followed by feature selection using Information Gain to reduce the feature space from nearly 13,000 to 753 informative features.

Overview of the multi-label text categorization process including preprocessing, training, and testing phases.

The study compared two primary ensemble strategies:

  1. Adaptive Boosting (AdaBoost.MH): An incremental approach that reweights difficult instances in successive training rounds to focus on errors.
  2. Bagging (Bagging.ML): A method that trains multiple classifiers on random subsets (bootstraps) of the data and aggregates their predictions via voting.

These strategies were paired with five base classifiers (Decision Stumps, J48, Random Forest, Naive Bayes, and SMO) and compared against Binary Relevance (BR) and Label Powerset (LP) baselines.

Results and Discussion

The effectiveness of ensemble methods was found to be highly dependent on the underlying base classifier. For instance, AdaBoost.MH excelled when paired with J48 but performed poorly with SMO.

Comparison of Hamming Loss and Subset Accuracy across different ensemble methods and iterations.

Key performance trends emerged from the evaluation:

  • Error Reduction: AdaBoost.MH with J48 achieved the lowest Hamming Loss, effectively reducing total misclassifications compared to the BR baseline.
  • Subset Accuracy: Bagging.ML-LP with SMO emerged as the top performer for Subset Accuracy, indicating it was most effective at correctly predicting the entire set of labels for a given text.
  • Minority Label Performance: A critical insight was that Bagging demonstrated greater potential for increasing the accuracy of minority labels compared to AdaBoost. While AdaBoost can sometimes over-focus on difficult majority instances, Bagging with SMO significantly improved accuracy across the board, including rare categories.

Conclusion

Ensemble methods offer a robust alternative to data-level resampling for handling imbalanced multi-label datasets. Bagging, in particular, shows promise for improving the visibility of minority classes in text categorization. However, these benefits come with increased computational costs due to the iterative nature of ensemble training. Future research may explore cost-sensitive learning or active learning to further refine the handling of the "long tail" in large-scale multi-label datasets.

Summary updated

BibTeX
@misc{winata2018handling,
  author = {Genta Indra Winata and Masayu Leylia Khodra},
  title = {{Handling Imbalanced Dataset in Multi-label Text Categorization using Bagging and Adaptive Boosting}},
  year = {2018},
  eprint = {1810.11612},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.11612},
}
Figure