IDIAP_TIET@LT-EDI-ACL2022 : Hope Speech Detection in Social Media using Contextualized BERT with Attention Mechanism

Deepanshu Khanna, Muskaan Singh, Petr MotlicekACL 2022

TL;DR

This paper tackles the crucial problem of automatically detecting "hope speech" on social media to combat the spread of hate speech. They propose a straightforward yet effective pipeline that leverages BERT's contextual embeddings and an attention mechanism, achieving a respectable fourth place in the HopeEDI shared task.

What Problem Does This Paper Address?

Social media platforms have become fertile ground for the rapid dissemination of hate speech, which can incite real-world violence and create a toxic online environment. While efforts are made to curb this, distinguishing hate speech from legitimate freedom of expression is a significant challenge, leading to potential false positives. This paper focuses on a related but distinct problem: identifying "hope speech." Hope speech, in this context, refers to content that promotes positivity, encouragement, and constructive dialogue, serving as a counter-narrative to hate speech. The authors aim to build a system that can automatically classify this positive content, contributing to a more balanced and less toxic online discourse.

Methods

The proposed method employs a two-stage pipeline:

  1. Contextualized BERT Embeddings: The authors utilize the RoBERTa-large-MNLI language model to generate contextualized embeddings for each comment. This is a standard approach in modern NLP, where BERT-based models excel at capturing the nuanced meaning of words based on their surrounding context. To handle comments longer than BERT's typical 512-token input limit, they employ a sliding window technique and average the resulting embeddings.
  2. Attention Mechanism: The generated embeddings are then fed into an attention network. This network, inspired by previous work, uses a Bi-LSTM layer to process the contextual representations. Non-linear activation functions (Tanh and Softmax) are applied to update training weights, allowing the model to focus on the most important parts of the comment for classification. Finally, a Dense layer with a sigmoid activation function performs the binary classification (Hope Speech vs. Not Hope Speech). This attention mechanism is key to filtering out less relevant information and focusing on discriminative features.

Datasets

The primary dataset used is the HopeEDI (Hope Speech Detection for Equality, Diversity, and Inclusion) shared task dataset. This dataset consists of YouTube comments in multiple languages, including English, Kannada, Malayalam, Spanish, and Tamil. The authors specifically focus on the English language comments for their experiments. The dataset is divided into training, development, and test sets, with a total of 22,740 training comments for English.

Key Insights and Ideas

  • Simplicity can be effective: The authors demonstrate that a relatively simple pipeline combining established techniques like BERT and attention can yield competitive results, even against more complex architectures.
  • Attention is crucial for context: The attention mechanism's ability to focus on salient parts of a comment is vital for accurately distinguishing between subtle forms of hope speech and other content.
  • Hope speech as a counter-narrative: The paper highlights the importance of identifying hope speech not just as a positive category but as a crucial tool to combat the pervasive issue of hate speech online.

Key Results

The proposed model achieved a weighted F1-score of 0.84 on the English dataset, ranking fourth in the HopeEDI shared task. For the validation set, the model achieved an F1-score of 0.80. As a point of comparison, a baseline model that fine-tuned RoBERTa-large-MNLI directly (without the attention mechanism) achieved an F1-score of 0.77 on the validation set, underscoring the benefit of the attention layer.

Limitations and Future Work

  • Language Focus: The current work focuses solely on English comments. The authors acknowledge this limitation and express their intention to extend their work to other languages present in the HopeEDI dataset.
  • Model Complexity: While the proposed model is relatively simple, future work could explore more complex neural network architectures.
  • Broader Application: The authors suggest exploring different neural network architectures combined with contextual embeddings for future research.

Summary updated

BibTeX
@inproceedings{khanna2022idiap,
  author = {Deepanshu Khanna and Muskaan Singh and Petr Motlicek},
  title = {{IDIAP\_TIET@LT-EDI-ACL2022 : Hope Speech Detection in Social Media using Contextualized BERT with Attention Mechanism}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure