cs_english@LT-EDI-EACL2021: Hope Speech Detection Based On Fine-tuning ALBERT Model

Shi Chen, Bing Kong2021

TL;DR

This paper tackles the crucial task of detecting "hope speech" on social media, aiming to foster more positive online environments. By fine-tuning the ALBERT model and employing K-fold cross-validation on an English dataset, the authors achieved a top-tier F1 score of 0.93, demonstrating the effectiveness of their approach.

What Problem Does This Paper Address?

In today's digital landscape, social media platforms often grapple with negativity, offensive content, and a general lack of inclusivity. While much research has focused on detecting and mitigating offensive speech, this paper highlights a less explored but equally important area: the detection of hope speech. The goal is to proactively identify and promote positive, encouraging content that can counter depression, confusion, and feelings of isolation, thereby cultivating a healthier and more supportive online community. The specific challenge is to accurately classify social media text into three categories: 'Hope speech', 'Not hope speech', and 'Not in intended language'.

Methods

The core of this work lies in leveraging a pre-trained language model, specifically ALBERT, and fine-tuning it for the hope speech detection task. ALBERT, a lighter variant of BERT, was chosen for its efficiency in training time and memory usage, while still offering strong performance. The authors employed a specific fine-tuning strategy: they combined the output of the last three layers of the ALBERT encoder with the special [CLS] token's output. This combined representation, shaped into a [batch_size, hidden_size*4] tensor, was then fed into a classifier. To ensure robust evaluation and mitigate overfitting, K-fold cross-validation was used. For the English task, a 5-fold cross-validation approach was implemented, where the data was split into five sets, with each set serving as a validation set once while the others were used for training.

Datasets

The authors utilized the English dataset provided by the LT-EDI 2021-EACL 2021 task organizers. This dataset was specifically curated for the "Hope Speech Detection for Equality, Diversity, and Inclusion" challenge. The dataset contains three categories: 'Hope speech', 'Non hope speech', and 'Not English'. The paper notes that the English dataset exhibits a significant class imbalance, with 'Non hope speech' constituting around 90% of the data.

Key Insights and Ideas

  • The Importance of Proactive Positivity: Moving beyond simply filtering out negative content, the paper emphasizes the value of actively identifying and promoting positive, hopeful messages to shape a better online environment.
  • ALBERT's Efficiency for Fine-tuning: The choice of ALBERT over BERT highlights a practical consideration for researchers: achieving high performance with reduced computational resources.
  • Strategic Feature Fusion: Combining outputs from multiple layers of the ALBERT encoder with the [CLS] token's representation proved to be an effective strategy for capturing nuanced linguistic information relevant to hope speech.
  • Class Imbalance as a Challenge: The significant imbalance in the English dataset (90% 'Non hope speech') underscores a common hurdle in real-world NLP tasks and the need for robust evaluation methods like cross-validation.

Key Results

The model achieved a strong performance on the English hope speech detection task. On the test set, the model attained a Precision of 0.93, Recall of 0.94, and an F1 Score of 0.93. This F1 score resulted in the team tying for first place on the task leaderboard. The paper also presents a breakdown of performance on the labeled test set, showing a high F1 score for 'Non hope speech' (0.97) and a lower, but still respectable, F1 score for 'Hope speech' (0.60).

Limitations and Future Work

The authors acknowledge that while their results are promising, they aim to continue improving their methods. They suggest exploring other models in future work to potentially achieve even better results on similar tasks. The paper also implicitly points to the challenge of class imbalance as an area that might warrant further investigation or specific handling techniques in future iterations.

Summary updated

Abstract

This paper mainly introduces the relevant content of the task “Hope Speech Detection for Equality, Diversity, and Inclusion at LT-EDI 2021-EACL 2021”. A total of three language datasets were provided, and we chose the English dataset to complete this task. The specific task objective is to classify the given speech into ‘Hope speech’, ‘Not Hope speech’, and ‘Not in intended language’. In terms of method, we use fine-tuned ALBERT and K fold cross-validation to accomplish this task. In the end, we achieved a good result in the rank list of the task result, and the final F1 score was 0.93, tying for first place. However, we will continue to try to improve methods to get better results in future work.
BibTeX
@misc{chen2021cs,
  author = {Shi Chen and Bing Kong},
  title = {{cs\_english@LT-EDI-EACL2021: Hope Speech Detection Based On Fine-tuning ALBERT Model}},
  year = {2021},
}
Figure