TEAM HUB@LT-EDI-EACL2021: Hope Speech Detection Based On Pre-trained Language Model

Bo Huang, Yang BaiLT-EDI@EACL 2021

TL;DR

This paper presents a system for detecting "hope speech" in multilingual social media comments, a crucial task for fostering positive online environments. The team achieved top rankings by fine-tuning the XLM-RoBERTa model and combining its contextual understanding with TF-IDF for improved robustness against code-mixing.

What Problem Does This Paper Address?

The proliferation of negative content, including hate speech and misinformation, on social media platforms is a significant concern. While efforts have been made to detect and mitigate such content, there's a growing recognition of the need to also identify and promote positive discourse. "Hope speech" – comments that convey positivity, courage, and promote harmony – is vital for a healthy online ecosystem. However, detecting this type of speech, especially in multilingual contexts with code-mixing (where multiple languages are used within a single post), presents a significant challenge. Existing systems often focus on languages with abundant resources, leaving a gap in addressing less-resourced languages and the complexities of mixed-language content.

Methods

The core of the TEAM HUB system is the XLM-RoBERTa pre-trained language model. This model is chosen for its strong multilingual capabilities, having been trained on over 100 languages, making it well-suited for the task's multilingual nature. The authors fine-tune XLM-RoBERTa on the hope speech dataset.

To address the challenge of code-mixing, they introduce a hybrid approach that combines the contextual embeddings from XLM-RoBERTa with TF-IDF (Term Frequency-Inverse Document Frequency). TF-IDF is a traditional method that captures the importance of words in a document relative to a corpus. The output of TF-IDF is weighted and then combined with the output of XLM-RoBERTa's last layer. This combined representation is then fed into an Inception block. Inception blocks, known for their effectiveness in capturing multi-scale features, are used here with convolution kernels to learn semantic information of varying lengths. This approach aims to leverage the strengths of both deep contextual understanding and keyword importance, while the Inception module helps in processing diverse semantic patterns. Finally, the output of the Inception module is passed to a classifier for prediction.

Datasets

The paper utilizes the dataset provided for the LT-EDI 2021: Hope Speech Detection Shared Task. This dataset consists of comments and posts from YouTube and is annotated for three languages: English, Malayalam, and Tamil. The dataset includes three label categories:

  • Hope_speech: Comments expressing positive sentiments, courage, interpersonal skills, beauty, perseverance, forgiveness, tolerance, future consciousness, talent, and wisdom.
  • Non_Hope_speech: Comments containing prejudices, attacks, anti-humanity, racial discrimination, and other negative content.
  • Not_English, Not_Malayalam, Not_Tamil: Comments in languages other than the primary language of the dataset (e.g., Tamil comments in the English dataset).

The authors note that the data exhibits significant class imbalance, with "Non_Hope_speech" being the dominant class. They also highlight the presence of code-mixing within the dataset.

Key Insights and Ideas

  • Multilingual Models are Key for Diverse Online Content: The success of XLM-RoBERTa underscores the importance of pre-trained models that can handle multiple languages simultaneously for tasks involving global social media.
  • Hybrid Approaches Combat Code-Mixing: Simply relying on a powerful multilingual model isn't enough. Combining its deep contextual understanding with a more traditional, keyword-focused method like TF-IDF proved effective in mitigating the negative impact of code-mixing.
  • Inception Blocks for Richer Feature Extraction: Using Inception blocks to process the combined XLM-RoBERTa and TF-IDF features allows the model to capture semantic information at different scales, which is beneficial for understanding the nuances of social media text.
  • Data Imbalance is a Persistent Challenge: The paper explicitly mentions the significant imbalance in the dataset, which is a common issue in real-world NLP tasks and requires careful consideration during model training and evaluation.

Key Results

The TEAM HUB system achieved impressive results in the LT-EDI 2021 Shared Task:

  • English Task: Ranked 1st, with an F1 score of 0.93.
  • Malayalam Task: Ranked 2nd, with an F1 score of 0.84.
  • Tamil Task: Ranked 3rd, with an F1 score of 0.59.

The authors note that their model's performance on the validation set was largely consistent with the test set results, except for a slight gap in the Tamil dataset.

Limitations and Future Work

The authors acknowledge that their data preprocessing was not extensively detailed. They specifically mention that they ignored the influence of stop words, emoticons, and other special symbols on the results, which they identify as a flaw.

For future work, they plan to:

  • Address the shortcomings in data preprocessing by incorporating more sophisticated techniques to handle stop words, emoticons, and special symbols.
  • Improve their methods based on the identified issues, particularly concerning the Tamil language task.
  • Continue to monitor advancements in low-resource language communities and the field of hope speech detection in natural language processing.

Summary updated

BibTeX
@inproceedings{huang2021team,
  author = {Bo Huang and Yang Bai},
  title = {{TEAM HUB@LT-EDI-EACL2021: Hope Speech Detection Based On Pre-trained Language Model}},
  booktitle = {LT-EDI@EACL},
  year = {2021},
}
Figure