IIITSurat@LT-EDI-ACL2022: Hope Speech Detection using Machine Learning

Pradeep Roy, Snehaan Bhawal, Abhinav Kumar, Bharathi Raja ChakravarthiACL 2022

TL;DR

This paper tackles the challenge of automatically detecting "hope speech" – comments expressing optimism and motivation – on social media. They developed a machine learning model that outperforms deep learning approaches, achieving a weighted F1-score of 0.55 and winning first place in the LT-EDI-ACL2022 competition for English.

What Problem Does This Paper Address?

Social media platforms are awash with user-generated content, ranging from personal achievements to pleas for help. Amidst this deluge, identifying "hope speech" – comments that convey positivity, resilience, and motivation – is crucial for understanding user sentiment and fostering supportive online environments. However, this is a challenging task due to the inherent imbalance in class distribution (far more non-hope speech than hope speech) and the subtle nature of hopeful expressions. Existing research has paid less attention to hope speech detection compared to other sentiment analysis tasks like hate speech detection.

Methods

The researchers explored both traditional machine learning (ML) models and deep learning (DL) techniques. For ML, they experimented with Logistic Regression (LR), Random Forest (RF), Naive Bayes (NB), and Extreme Gradient Boosting (XGB). To address the class imbalance, they employed three oversampling techniques: Random Oversampling (ROS), SMOTE (Synthetic Minority Over-sampling Technique), and ADASYN (Adaptive Synthetic Sampling). Feature extraction was done using TF-IDF with 1-5 n-grams. For deep learning, they implemented DNN, DNN with embeddings, CNN, LSTM, and BiLSTM models.

Datasets

The study utilized a dataset compiled from YouTube comments for the LT-EDI-ACL2022 workshop's "Hope Speech Detection for Equality, Diversity, and Inclusion" task. This dataset was available for three languages: Tamil, Malayalam, and English. The research specifically focused on the English portion, which was labeled into two categories: 'Hope Speech' and 'Non-Hope Speech'. The training dataset contained 1,962 hope speech samples and 20,778 non-hope speech samples, while the validation set had 272 hope speech and 2,569 non-hope speech samples. The test dataset was unlabeled.

Key Insights and Ideas

  • Class Imbalance is a Major Hurdle: The significant disparity between hope and non-hope speech samples makes it difficult for models to learn the minority class effectively. Oversampling techniques are essential for creating a more balanced training set.
  • Traditional ML Shines on Balanced Data: Surprisingly, traditional ML models, particularly Random Forest, outperformed deep learning models when trained on the balanced dataset. This suggests that for this specific task and dataset, simpler models can be more effective when properly trained.
  • SMOTE is the Champion Oversampler: Among the oversampling techniques, SMOTE consistently yielded better results than ADASYN and ROS, indicating its effectiveness in generating synthetic samples for the minority class without introducing excessive noise or overfitting.
  • Random Forest's Strength: The Random Forest classifier, when combined with SMOTE oversampling, demonstrated robust performance, achieving high accuracy and F1-scores.

Key Results

The proposed automated machine learning model, specifically using Random Forest with SMOTE oversampling, achieved a weighted F1-score of 0.55 on the test dataset. This performance secured the first rank among the participating teams in the English language category of the LT-EDI-ACL2022 competition. The study found that ML models generally performed better than DL models on the balanced dataset. For instance, the RF model achieved an F1-score of 0.92, outperforming DNN (0.91), DNN+Emb (0.83), CNN (0.87), LSTM (0.86), and BiLSTM (0.87) on a balanced dataset.

Limitations and Future Work

The authors acknowledge that the model's performance can be further improved. They suggest exploring more advanced preprocessing steps and expanding the feature set. Additionally, they propose investigating transformer-based models and ensemble methods as promising avenues for future research to potentially achieve even better performance in hope speech detection.

Summary updated

BibTeX
@inproceedings{roy2022iiitsurat,
  author = {Pradeep Roy and Snehaan Bhawal and Abhinav Kumar and Bharathi Raja Chakravarthi},
  title = {{IIITSurat@LT-EDI-ACL2022: Hope Speech Detection using Machine Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure