bitsa_nlp@LT-EDI-ACL2022: Leveraging Pretrained Language Models for Detecting Homophobia and Transphobia in Social Media Comments

Vitthal Bhandari, Poonam GoyalACL 2022

TL;DR

This paper tackles the crucial but under-researched problem of detecting homophobia and transphobia in social media comments, particularly in low-resource languages like Tamil and in code-mixed text. They demonstrate that pre-trained multilingual transformer models, like mBERT, can achieve surprisingly good performance even without explicit code-mixing pre-training, highlighting the power of cross-lingual transfer learning.

What Problem Does This Paper Address?

Online social media platforms are rife with hate speech and cyberbullying, yet specific forms of discrimination, such as homophobia and transphobia, remain under-addressed in research. This is particularly true for non-English languages, where annotated datasets are scarce, and for code-mixed text (where users switch between languages within a single comment), which poses a significant challenge for standard NLP models. The authors aim to bridge this gap by developing and evaluating systems for detecting these specific forms of hate speech in YouTube comments, focusing on English and Tamil, and their code-mixed variants.

Methods

The core of the system relies on leveraging pre-trained multilingual transformer models. The authors experimented with several popular models, including BERT, mBERT, XLM-ROBERTa, IndicBERT, and HateBERT. These models were fine-tuned on a dataset of YouTube comments labeled for homophobia and transphobia.

To address the significant class imbalance in the dataset (where homophobia and transphobia are minority classes), they employed a data augmentation technique called Surface Form Alteration (EDA). EDA involves randomly deleting, inserting, or swapping words, and performing synonym replacement. For the English dataset, EDA was applied to oversample the minority classes. Downsampling was used for the majority class (Non-anti-LGBT+ content) to further mitigate imbalance.

Preprocessing steps included removing punctuation and de-emojifying comments by replacing emojis with their English textual descriptions.

Datasets

The primary dataset used in this study is a collection of 15,141 multilingual YouTube comments, specifically annotated for homophobia, transphobia, or non-anti-LGBT+ content. This dataset was provided by the organizers of the LT-EDI shared task. The dataset was split into English, Tamil, and Tamil-English (code-mixed) subtasks.

Key Insights and Ideas

  • Multilingual Models Shine: mBERT, a multilingual transformer model, emerged as the top performer across all three tracks (English, Tamil, and code-mixed), even outperforming models specifically trained on hate speech (HateBERT) or Indian languages (IndicBERT) in some cases.
  • Zero-Shot Cross-Lingual Transfer is Powerful: The success of mBERT on the Tamil and code-mixed tasks, without explicit pre-training on these specific scenarios, demonstrates the remarkable capability of these models for zero-shot cross-lingual transfer.
  • Data Augmentation's Nuance: While EDA was used to combat class imbalance, its effectiveness was not uniform. It proved beneficial for the English dataset but did not significantly improve performance on Tamil or code-mixed tasks, suggesting that the optimal data augmentation strategy is task and language-dependent.
  • Preprocessing's Mixed Bag: De-emojification and de-punctuation had a mixed impact. While they slightly reduced performance on English and Tamil, they showed a significant improvement for the code-mixed task, indicating that handling informal elements like emojis is crucial for complex linguistic scenarios.

Key Results

The authors' submission achieved impressive rankings in the LT-EDI shared task:

  • English: Rank 9 with a macro-averaged F1-score of 0.42.
  • Tamil: Rank 6 with a macro-averaged F1-score of 0.64.
  • Tamil-English (Code-mixed): Rank 3 with a macro-averaged F1-score of 0.58.

Specifically, mBERT achieved the following macro-averaged F1-scores on the test sets:

  • English: 0.42
  • Tamil: 0.72
  • Tamil-English: 0.58

The paper also highlights that a simple majority classifier, without any class imbalance handling, performed poorly with macro-averaged F1-scores around 0.30 across all tracks, underscoring the importance of their methods.

Limitations and Future Work

The authors acknowledge that the effectiveness of preprocessing varied across languages, and that EDA, while useful, is not a universally optimal data augmentation technique. They also note that transformer models performed well even with limited samples in the absence of augmentation, suggesting that the dataset size might have been sufficient for these powerful models.

Future work proposed includes exploring more aggressive data augmentation techniques, such as text generation (text in-filling, generating typos) or using auxiliary datasets. They also plan to investigate the impact of translation and transliteration on code-mixed text classification.

Summary updated

BibTeX
@inproceedings{bhandari2022bitsa,
  author = {Vitthal Bhandari and Poonam Goyal},
  title = {{bitsa\_nlp@LT-EDI-ACL2022: Leveraging Pretrained Language Models for Detecting Homophobia and Transphobia in Social Media Comments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure