Semantic and Contextual Modeling for Malicious Comment Detection with BERT-BiLSTM

Zhou Fang, Hanlu Zhang, Jacky He, Zhen Qi, Hongye ZhengarXiv 2025 · DOI · arXiv

Abstract

The rapid growth of social media has necessitated robust automated systems for detecting malicious comments. This paper proposes a hybrid architecture combining Bidirectional Encoder Representations from Transformers (BERT) with Bidirectional Long Short-Term Memory (BiLSTM) networks. While BERT captures deep semantic nuances, the BiLSTM layer models long-range sequential dependencies. Evaluated on the Jigsaw toxicity dataset, the BERT-BiLSTM model achieves a superior accuracy of 0.94, demonstrating significant improvements over standalone BERT and traditional machine learning baselines.

Introduction

As digital communication platforms expand, the prevalence of malicious content—including hate speech, harassment, and misinformation—poses a significant challenge to community safety. Detecting such content is difficult due to linguistic subtlety, where toxic intent is often masked by context, and the inherent data imbalance where malicious comments are rare compared to benign ones. Traditional methods like TF-IDF or Support Vector Machines often fail to capture the complex semantics required for accurate detection.

Methodology: The BERT-BiLSTM Architecture

The proposed solution utilizes a "dual-model" approach to maximize both semantic understanding and sequential context.

BERT Feature Extraction

BERT serves as the primary feature extractor. Pre-trained on massive corpora, it excels at understanding polysemy—the way word meanings shift based on surrounding context. The model extracts feature vectors from BERT’s final layer, specifically utilizing the [CLS] token to represent the semantic essence of the entire sentence.

BiLSTM Sequential Modeling

To complement BERT, a BiLSTM layer is integrated to process the extracted features. While BERT handles semantics, the BiLSTM captures temporal dynamics and long-term dependencies by processing text in both forward and backward directions. This ensures that the relationship between words at opposite ends of a comment is preserved, mitigating the "gradient vanishing" problem common in standard RNNs.

Architecture of the proposed BERT-BiLSTM hybrid model

The outputs from BERT are weighted and fed into the BiLSTM, followed by a Softmax activation function to produce a final probability score for binary classification.

Experimental Setup

The model was evaluated using the Jigsaw Unintended Bias in Toxicity Classification dataset. This dataset, hosted on Kaggle, contains a large volume of public comments labeled by human raters. It is specifically curated to help researchers develop models that recognize toxicity while minimizing bias against specific identity groups.

Results and Discussion

The BERT-BiLSTM model was benchmarked against several baselines, including traditional machine learning (TF-IDF + Random Forest) and other deep learning architectures (TextCNN, TextRNN, and standalone BERT).

Performance comparison of the BERT-BiLSTM model against various baseline methods

As shown in the results, the hybrid model outperformed all baselines. Notably, it achieved a recall of 0.93 compared to standalone BERT's 0.91, indicating a superior ability to identify malicious comments that might otherwise be missed. The synergy between "deep" semantic features and "long" sequential modeling allows the model to handle complex structures like sarcasm or subtle insults more effectively than single-architecture approaches.

Conclusion

The BERT-BiLSTM hybrid model represents a significant step forward in automated content moderation. By capturing both the "what" (semantics) and the "how" (structure) of malicious comments, it provides a more robust detection mechanism. Future research will focus on reducing the computational complexity of the model for real-time applications and enhancing the explainability of the classification decisions to ensure transparent and fair moderation.

Summary updated

BibTeX
@misc{fang2025semantic,
  author = {Zhou Fang and Hanlu Zhang and Jacky He and Zhen Qi and Hongye Zheng},
  title = {{Semantic and Contextual Modeling for Malicious Comment Detection with BERT-BiLSTM}},
  year = {2025},
  eprint = {2503.11084},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.11084},
}
Figure