Overview of The Shared Task on Homophobia and Transphobia Detection in Social Media Comments

Bharathi Raja Chakravarthi, Ruba Priyadharshini, Durairaj Thenmozhi, John Philip McCrae, Paul Buitelaar, Rahul Ponnusamy, Prasanna Kumar KumaresanACL 2022

TL;DR

This paper presents an overview of a shared task focused on detecting homophobia and transphobia in social media comments across Tamil, English, and Tamil-English (code-mixed) languages. It highlights the creation of a new dataset and the performance of various machine learning and deep learning models, underscoring the ongoing challenges and opportunities in this critical area of NLP.

What Problem Does This Paper Address?

Social media has become a primary platform for communication, but it also harbors toxic content, including hate speech directed at LGBTQ+ individuals. Homophobia and transphobia are significant forms of this hate speech, leading to real-world harm and discrimination. While research exists for detecting such content in English, there's a critical need for similar capabilities in under-resourced languages like Tamil, especially in code-mixed contexts where languages are blended. This paper addresses this gap by detailing a shared task designed to foster research and develop robust models for detecting homophobia and transphobia in these diverse linguistic settings.

Methods

The shared task involved participants developing systems to classify social media comments into three categories: "Homophobic," "Transphobic," and "Non-anti-LGBT+ content." The methodologies employed by the participating teams were diverse, but a common theme was the utilization of pre-trained language models, particularly transformer-based architectures. These included models like BERT, mBERT, XLM-ROBERTa, and IndicBERT, often fine-tuned on the provided datasets. Some teams also incorporated traditional machine learning classifiers (e.g., SVM, Random Forest) and feature engineering techniques like TF-IDF and word embeddings. Ensemble methods, such as majority voting and weighted voting, were also explored to combine the predictions of multiple models.

Datasets

The core dataset for this shared task was newly created and specifically designed for homophobia and transphobia identification. It comprises social media comments in three languages:

  • Tamil: A Dravidian language with a rich history and millions of speakers.
  • English: The widely used global language.
  • Tamil-English (code-mixed): Reflecting the common practice of blending these languages in online communication.

The dataset was collected from YouTube comments and manually annotated by members of the LGBTQ+ community and their allies, ensuring a nuanced and accurate labeling process. The paper notes that the dataset exhibits class imbalance, with "Non-anti-LGBT+ content" being the majority class across all language settings.

Key Insights and Ideas

  • The Importance of Code-Mixed Language Processing: The inclusion of Tamil-English code-mixed data is crucial, as it reflects the reality of online communication in many regions and presents unique challenges for NLP models.
  • Transformer Models Dominate: Pre-trained transformer models, especially XLM-ROBERTa, proved highly effective, even in zero-shot settings, highlighting their power in handling data imbalance and multilingualism.
  • Class Imbalance is a Persistent Challenge: The inherent imbalance in the dataset (more non-hate speech than hate speech) requires careful consideration and specific techniques (like macro-averaging for evaluation) to ensure fair model performance across all classes.
  • Dravidian NLP is Growing: The shared task itself signifies a growing interest and effort in developing NLP technologies for Dravidian languages, which are often under-resourced.

Key Results

The shared task saw submissions from 14 teams across the three language settings. The best-performing systems achieved macro F1-scores of 0.570 for Tamil, 0.870 for English, and 0.610 for Tamil-English. The Ablimet team, using a fine-tuned XLM-ROBERTa model, achieved the top performance in the English dataset. The ARGUABLY team demonstrated strong performance in both Tamil and Tamil-English tasks, leveraging machine learning and deep learning architectures. While English performance was generally higher, the results indicate significant room for improvement in Tamil and Tamil-English detection.

Limitations and Future Work

The authors acknowledge that the performance across all languages, particularly Tamil and Tamil-English, still has considerable room for improvement. They also note the class imbalance in the dataset as a challenge that was addressed through evaluation metrics like macro-averaged F1-score.

Future work suggested by the authors includes:

  • Expanding the effort to include more Dravidian languages like Malayalam and Kannada.
  • Developing systems capable of real-time detection.
  • Incorporating mixed-script data to further enhance robustness.

Summary updated

BibTeX
@inproceedings{chakravarthi2022overview,
  author = {Bharathi Raja Chakravarthi and Ruba Priyadharshini and Durairaj Thenmozhi and John Philip McCrae and Paul Buitelaar and Rahul Ponnusamy and Prasanna Kumar Kumaresan},
  title = {{Overview of The Shared Task on Homophobia and Transphobia Detection in Social Media Comments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure