IIT Dhanbad @LT-EDI-ACL2022- Hope Speech Detection for Equality, Diversity, and Inclusion

Vishesh Gupta, Ritesh Kumar, Rajendra PamulaACL 2022

TL;DR

This paper tackles the crucial task of automatically detecting "hope speech" on social media, aiming to promote positive and supportive content. The researchers developed and evaluated several machine learning models, with XGBoost achieving the best performance in classifying comments as either hope speech or non-hope speech. This work contributes to building more inclusive and positive online environments by providing tools to identify and amplify constructive discourse.

What Problem Does This Paper Address?

Social media has become a double-edged sword, offering connectivity but also serving as a breeding ground for hate speech and cyberbullying. While platforms strive to moderate harmful content, the sheer volume of data makes manual detection challenging. This paper focuses on a specific aspect of positive online discourse: "hope speech." Hope speech is defined as content that offers support, reassurance, suggestions, inspiration, and insight, reflecting a belief in achieving desired objectives. The challenge lies in automatically identifying this positive content to counteract the prevalence of negativity and discrimination online, thereby fostering equality, diversity, and inclusion.

Methods

The researchers employed a range of established machine learning algorithms for text classification. Their approach involved:

  • Data Preprocessing: Standard text cleaning techniques were applied, including removing punctuation, links, and numbers, converting text to lowercase, tokenization, and padding sequences to a uniform length.
  • Handling Imbalanced Data: Recognizing that hope speech might be less frequent than non-hope speech, they utilized the Synthetic Minority Oversampling Technique (SMOTE) to balance the training dataset.
  • Machine Learning Models: The core of their methodology involved training and evaluating several classifiers:
  • Logistic Regression
    • Multinomial Naive Bayes
    • Random Forest
    • XGBoost

These models were implemented using the scikit-learn library, with specific hyperparameter tuning for Random Forest (n_estimators=1000, random_state=42) and XGBoost (learning_rate=0.01, max_depth=50, n_estimators=300).

Datasets

The study utilized a dataset provided for the Hope Speech Detection for Equality, Diversity, and Inclusion challenge at LT-EDI-ACL 2022. This dataset consists of YouTube comments labeled as either "Hope Speech" (HS) or "Non Hope Speech" (NHS). The paper details the distribution of this data across training, development, and test sets, noting the imbalanced nature of the classes in the training data.

Key Insights and Ideas

  • The Importance of Hope Speech: The paper highlights that hope speech is not just about avoiding negativity but actively contributing to well-being, recuperation, and motivation. This positive framing is crucial for fostering a healthier online environment.
  • XGBoost's Superiority: Among the tested models, XGBoost consistently outperformed others, suggesting its effectiveness in capturing the nuances of hope speech in social media text.
  • Challenges of Imbalanced Data: The authors explicitly address the issue of imbalanced classes, which is a common challenge in real-world classification tasks. Their use of SMOTE demonstrates a practical approach to mitigate this.
  • Potential for Broader Impact: By developing tools to detect hope speech, the research opens avenues for platforms to promote positive content, potentially influencing user behavior and online community health.

Key Results

The primary quantitative result is the performance of the models on the test dataset. The best score achieved was a Macro-F1 of 0.6130, obtained by the XGBoost model. The paper also presents accuracy, weighted-F1, and micro-F1 scores for all tested algorithms, with XGBoost generally showing strong performance across these metrics. The authors note that the imbalance between classes in the test data contributed to differences between accuracy and Macro-F1 scores.

Limitations and Future Work

The authors acknowledge that their current best score is 0.6130 and express a desire to experiment with more advanced algorithms or neural network models. They also plan to explore random multi-model classification for potentially better accuracy and classification. Furthermore, fine-tuning the parameters of existing algorithms is identified as a promising avenue for improving overall performance. The paper suggests that these tasks will be explored in the coming days.

Summary updated

BibTeX
@inproceedings{gupta2022iit,
  author = {Vishesh Gupta and Ritesh Kumar and Rajendra Pamula},
  title = {{IIT Dhanbad @LT-EDI-ACL2022- Hope Speech Detection for Equality, Diversity, and Inclusion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure