MuTCELM: An optimal multi-TextCNN-based ensemble learning for text classification

Victor Kwaku Agbesi, Wenyu Chen, Sophyani Banaamwini Yussif, Chiagoziem C. Ukwuoma, Yeong Hyeon Gu, Mugahed A. Al–antariHeliyon 2024 · DOI

TL;DR

This paper introduces MuTCELM, a novel ensemble learning model that significantly improves text classification performance, especially for complex and low-resource languages. By combining a new feature extraction technique (Multi-TextCNN) with multiple transformer-based sub-classifiers, MuTCELM achieves state-of-the-art results across various datasets, demonstrating its robustness and effectiveness.

What Problem Does This Paper Address?

Text classification, the task of categorizing text into predefined classes, is crucial for many applications, from recommendation systems to sentiment analysis. However, accurately classifying text is challenging due to the inherent complexity of language, including polysemy (words with multiple meanings), contextual nuances, and variations across different dialects and languages. Traditional methods often struggle to capture these complexities, leading to suboptimal performance. Furthermore, many existing models are computationally expensive and don't generalize well to diverse linguistic structures, particularly for languages with limited available data (low-resource languages) like Ewe, Arabic, and Urdu. This paper aims to address these limitations by developing a more effective and robust text classification model.

Methods

The core innovation of this work is the Multi-TextCNN module, a novel feature extractor designed to capture textual information at multiple granularities. Unlike standard CNNs that might focus on local patterns, Multi-TextCNN uses convolutional kernels of varying sizes (specifically [3, 5, 7]) to extract features that represent trigrams, five-grams, and seven-grams, thereby capturing both local and longer-range dependencies.

This Multi-TextCNN module is then integrated into an ensemble learning framework called MuTCELM. MuTCELM constructs five distinct sub-classifiers by combining the Multi-TextCNN with different pre-trained transformer models (ALBERT, BERT, DeBERTa, Transformer-XL, and XLM-ROBERTa). Each sub-classifier leverages the strengths of its respective transformer model and the Multi-TextCNN to extract unique textual features. The outputs from these five sub-classifiers are then combined using a weighted voting ensemble approach, where the weights are determined by each sub-classifier's performance. This ensemble strategy aims to leverage the complementary strengths of the sub-classifiers, leading to improved overall accuracy and robustness.

Datasets

The study evaluates MuTCELM on datasets from three distinct languages:

  • Ewe: News samples from Nigeria, Benin, Togo, and Ghana, categorized into six classes.
  • Arabic:
    • AJGT (Arabic Jordanian General Tweets): Tweets annotated for sentiment (positive/negative).
  • ALJ-News (Al-Jazeera-News): News articles from aljazeera.net, categorized into five classes.
  • ArSarcasm-V2: Tweets annotated for sarcasm, encompassing multiple Arabic dialects.
  • Urdu Corpus: Articles for fake news detection.

These datasets were chosen to test the model's multilingual capabilities and its performance on languages with varying complexities and resource availability.

Key Insights and Ideas

  • Multi-granularity feature extraction is key: The Multi-TextCNN's ability to capture features at different n-gram levels (trigrams, five-grams, seven-grams) is crucial for understanding complex linguistic patterns and relationships within text.
  • Ensembling diverse models boosts performance: Combining multiple transformer-based models, each with its own strengths, within an ensemble framework significantly outperforms individual models.
  • Weighted voting is effective: Assigning weights to sub-classifiers based on their performance in the ensemble ensures that more reliable models have a greater influence on the final prediction.
  • Robustness across languages: MuTCELM demonstrates strong performance not only on widely spoken languages like Arabic but also on lower-resource languages like Ewe, highlighting its adaptability.
  • Statistical significance matters: The non-overlapping confidence intervals and statistical significance tests confirm that MuTCELM's performance gains are not due to random chance, underscoring its superiority.

Key Results

MuTCELM consistently outperformed baseline models and models enhanced with Multi-TextCNN across all evaluated datasets and metrics (accuracy, precision, recall, and F1-macro).

  • Average improvements: The paper reports average improvements across all datasets of 0.2584 in accuracy, 0.2546 in precision, 0.2668 in recall, and 0.2612 in F1-macro compared to baseline models.
  • Specific dataset highlights:
    • On the AJGT dataset, MuTCELM achieved an accuracy of 0.930, outperforming the best baseline (ELM) by 7.01%.
  • For ALJ-News, MuTCELM reached 0.949 accuracy.
  • On ArSarcasm-V2, MuTCELM achieved 0.957 accuracy, a significant improvement over previous methods.
  • The Ewe dataset saw the highest accuracy for MuTCELM at 0.977.
  • For the Urdu Corpus, MuTCELM achieved 0.913 accuracy.
  • Computational efficiency: While MuTCELM has higher training and inference times compared to some simpler models, the paper argues this is justified by its superior performance and that the inference time difference is less pronounced, making it viable for real-time applications.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that MuTCELM, due to its ensemble nature and the use of multiple transformer models, incurs higher computational costs for training and inference compared to simpler models.
  • Data Dependency: The model's performance can vary based on data-specific characteristics, highlighting the importance of considering linguistic diversity.
  • Future Work: The authors suggest expanding MuTCELM's application to a broader range of languages and domains. They also propose exploring additional features like contextual embeddings and transfer learning to further enhance model performance. Additionally, leveraging more powerful hardware (GPUs, TPUs) is suggested to mitigate computational costs.

Summary updated

Abstract

Feature extraction plays a critical role in text classification, as it converts textual data into numerical representations suitable for machine learning models. A key challenge lies in effectively capturing both semantic and contextual information from text at various levels of granularity while avoiding overfitting. Prior methods have often demonstrated suboptimal performance, largely due to the limitations of the feature extraction techniques employed. To address these challenges, this study introduces Multi-TextCNN, an advanced feature extractor designed to capture essential textual information across multiple levels of granularity. Multi-TextCNN is integrated into a proposed classification model named MuTCELM, which aims to enhance text classification performance. The proposed MuTCELM leverages five distinct sub-classifiers, each designed to capture different linguistic features from the text data. These sub-classifiers are integrated into an ensemble framework, enhancing the overall model performance by combining their complementary strengths. Empirical results indicate that MuTCELM achieves average improvements across all datasets in accuracy, precision, recall, and F1-macro scores by 0.2584, 0.2546, 0.2668, and 0.2612, respectively, demonstrating significant performance gains over baseline models. These findings underscore the effectiveness of Multi-TextCNN in improving model performance relative to other ensemble methods. Further analysis reveals that the non-overlapping confidence intervals between MuTCELM and baseline models indicate statistically significant differences, suggesting that the observed performance improvements of MuTCELM are not attributable to random chance but are indeed statistically meaningful. This evidence indicates the robustness and superiority of MuTCELM across various languages and text classification tasks.
BibTeX
@inproceedings{agbesi2024mutcelm,
  author = {Victor Kwaku Agbesi and Wenyu Chen and Sophyani Banaamwini Yussif and Chiagoziem C. Ukwuoma and Yeong Hyeon Gu and Mugahed A. Al–antari},
  title = {{MuTCELM: An optimal multi-TextCNN-based ensemble learning for text classification}},
  booktitle = {Heliyon},
  year = {2024},
  doi = {10.1016/j.heliyon.2024.e38515},
}
Figure