MUCIC@LT-EDI-ACL2022: Hope Speech Detection using Data Re-Sampling and 1D Conv-LSTM

Anusha Gowda, Fazlourrahman Balouchzahi, Hosahalli Shashirekha, Grigori SidorovACL 2022

TL;DR

This paper presents a novel approach for detecting "hope speech" on social media, a crucial but under-researched area. By employing data re-sampling techniques to address class imbalance and a 1D Convolutional Neural Network with Long Short-Term Memory (1D Conv-LSTM), the authors achieved top performance in the English language category of the HopeEDI shared task.

What Problem Does This Paper Address?

Social media has become a significant platform for individuals to share their thoughts and emotions. While much research focuses on identifying negative content like hate speech or fake news, there's a growing need to understand and promote positive content that offers support, motivation, and hope. This "hope speech" can be vital for individuals navigating difficult times. The HopeEDI shared task aims to address this by classifying YouTube comments into "Hope," "Not-Hope," or "Not-Intended" categories across multiple languages, including English, Spanish, Tamil, Malayalam, and Kannada. A key challenge in this task is the inherent imbalance in the dataset, where "hope speech" is often a minority class, making it difficult for models to learn effectively.

Methods

The MUCIC team's methodology involves two main stages: pre-processing and model construction.

  • Pre-processing: This stage focuses on cleaning the text data. It involves removing punctuation, numerical data, stopwords, and common uninformative phrases. Uppercase characters are converted to lowercase to standardize the text. Crucially, to combat the class imbalance issue, the Synthetic Minority Oversampling Technique (SMOTE) is applied to the English and Kannada datasets. SMOTE generates synthetic samples for the minority class ("Hope" class) by interpolating between existing minority class samples and their nearest neighbors.
  • Model Construction: The pre-processed text is tokenized and converted into sequences. A vocabulary size of 15,000 and a maximum sequence length of 50 are used. The sequences are then padded to ensure uniform length. The core of the model is a 1D Convolutional Neural Network (CNN) integrated with a Long Short-Term Memory (LSTM) network (1D Conv-LSTM). This architecture leverages the feature extraction capabilities of CNNs and the sequential modeling power of LSTMs. The model uses Keras embedding layers, followed by convolutional layers with 64 filters, pooling layers, and a ReLU activation function. This is then fed into 100 fully connected LSTM layers, culminating in a softmax output layer for classification.

Datasets

The study utilizes the HopeEDI shared task dataset, which comprises YouTube comments in English, Spanish, Tamil, Malayalam, and Kannada. For this work, the focus is on English, Tamil, and Kannada texts. The dataset is divided into training, development, and test sets. The paper highlights that the test sets for Kannada and Malayalam contained additional "Not-Kannada" and "Not-Malayalam" labels not present in the training data, which impacted performance for these languages.

Key Insights and Ideas

  • The Power of Re-sampling: The application of SMOTE proved highly effective in addressing the class imbalance for the English dataset, significantly contributing to the model's success. This underscores the importance of data balancing techniques in NLP tasks with skewed class distributions.
  • Hybrid CNN-LSTM Architecture: The combination of CNN and LSTM layers effectively captures both local features and sequential dependencies within the text, proving to be a robust architecture for hope speech detection.
  • Language-Specific Performance: The model achieved excellent results for English but struggled with Kannada and Malayalam. This highlights the challenges of multilingual NLP, especially when dealing with code-mixed languages and potential discrepancies in dataset labeling across languages.
  • The Importance of "Hope" Content: The paper emphasizes the need to shift focus from solely detecting negative content to also identifying and promoting positive and supportive messages on social media.

Key Results

The proposed methodology achieved 1st rank in the HopeEDI shared task for the English language category.

  • For English texts, the model obtained a macro-averaged F1-score of 0.550 and a weighted-averaged F1-score of 0.860 on the development set.
  • On the test set, the English model achieved a macro-averaged F1-score of 0.550.
  • Performance for Kannada and Malayalam texts was significantly lower than expected, with macro-averaged F1-scores of 0.310 and 0.310 respectively on the test set. This was attributed to the presence of unseen "Not-Kannada" and "Not-Malayalam" labels in the test sets.

Limitations and Future Work

  • Limitations: The primary limitation identified is the model's poor performance on Kannada and Malayalam texts, largely due to the presence of unseen labels in the test sets that were not part of the training data. The paper also notes that the SMOTE technique was only applied to English and Kannada texts.
  • Future Work: The authors suggest extending their experiments to explore various feature types, such as stylistic features, and to investigate different word embeddings and language models. They also aim to address the challenges posed by multilingual and code-mixed languages more effectively.

Summary updated

BibTeX
@inproceedings{gowda2022mucic,
  author = {Anusha Gowda and Fazlourrahman Balouchzahi and Hosahalli Shashirekha and Grigori Sidorov},
  title = {{MUCIC@LT-EDI-ACL2022: Hope Speech Detection using Data Re-Sampling and 1D Conv-LSTM}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure