SSNCSE_NLP@LT-EDI-ACL2022: Homophobia/Transphobia Detection in Multiple Languages using SVM Classifiers and BERT-based Transformers

Krithika Swaminathan, Bharathi B, Gayathri G L, Hrishik SampathACL 2022

TL;DR

This paper tackles the crucial problem of detecting homophobic and transphobic comments on social media, particularly in English, Tamil, and code-mixed Tamil-English. The researchers explored various machine learning and transformer-based approaches, achieving promising results with SVM classifiers and BERT-based models, highlighting the effectiveness of TF-IDF and LaBSE for specific language contexts.

What Problem Does This Paper Address?

Online platforms, while offering a space for marginalized communities like LGBTQIA+ individuals to connect and express themselves, also unfortunately host a significant amount of toxic and hateful content. Homophobic and transphobic comments are prevalent, fueled by the anonymity social media provides. Manually identifying and filtering this harmful content is an overwhelming and often ineffective task. This paper addresses the challenge of automatically detecting and classifying such sentiments, with a specific focus on the complexities introduced by code-mixed languages, such as Tamil and English being used together in a single utterance.

Methods

The researchers employed a multi-pronged approach to tackle this classification task. They began with standard feature extraction techniques, including:

  • Count Vectorizer: This method tokenizes text and represents it as a vector of token counts, essentially counting how many times each word appears.
  • TF-IDF (Term Frequency-Inverse Document Frequency): This technique quantifies the importance of words in a document relative to a collection of documents. It gives higher scores to words that are frequent in a specific text but rare across the entire corpus, thus highlighting distinctive terms.

Following feature extraction, they utilized various classifiers:

  • SVM (Support Vector Machines): A robust and widely used algorithm for classification.
  • K-Nearest Neighbors (KNN): A simple yet effective instance-based learning algorithm.
  • MLP (Multi-Layer Perceptron): A type of artificial neural network.

In addition to traditional methods, the paper also explored the power of BERT-based transformer models, which are known for their advanced natural language understanding capabilities. Specifically, they experimented with models like LaBSE (Language-Agnostic BERT Sentence Embedding), Indo-Aryan XLM-R-Base, and monsoon-nlp/tamillion, which are designed to handle multilingual text.

Datasets

The study utilized three distinct datasets for training and evaluation, all sourced from social media platforms, primarily YouTube:

  • English Dataset: Containing comments in English.
  • Tamil Dataset: Containing comments in Tamil.
  • Tamil-English Code-Mixed Dataset: Containing comments that blend both Tamil and English.

These datasets were classified into three categories: Homophobic, Transphobic, and Non-anti-LGBT+ content. The paper notes that these datasets are relatively small and imbalanced, which presents a challenge for model training.

Key Insights and Ideas

  • Language-Specific Strengths: The research highlights that different models and feature extraction techniques perform best for different languages. What works well for English might not be optimal for Tamil or code-mixed text.
  • TF-IDF's Resilience: For Tamil text, the TF-IDF feature extraction method, when combined with SVM, showed strong performance, suggesting its effectiveness in capturing relevant linguistic nuances even in non-English contexts.
  • LaBSE for Code-Mixing: The LaBSE model emerged as a particularly strong performer for the Tamil-English code-mixed dataset, underscoring the value of language-agnostic models for handling multilingual and mixed-language inputs.
  • SVM's Consistent Performance: SVM classifiers, when paired with appropriate feature extraction methods, consistently delivered good results across various datasets, demonstrating their reliability.
  • Data Imbalance is a Hurdle: The inherent imbalance in the datasets (significantly more "Non-anti-LGBT+" content than hateful comments) is a significant challenge that can lead to biased model predictions.

Key Results

The paper reports the following key quantitative results:

  • English Dataset: Achieved a weighted F1-score of 0.93 using average word embeddings and an SVM classifier.
  • Tamil Dataset: Achieved a weighted F1-score of 0.75 using TF-IDF and an SVM classifier.
  • Tamil-English Code-Mixed Dataset: Achieved a weighted F1-score of 0.87 using the LaBSE transformer model.

Qualitatively, the study found that while models performed well on the English dataset, there was more variation in performance for the Tamil and code-mixed datasets. The SVM and MLP classifiers showed similar accuracy rates, with SVM having a slight edge. Sentence transformers pre-trained for multilingual texts were found to be particularly effective for Tamil.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Size and Imbalance: The datasets used are relatively small and imbalanced, which can affect model generalization and introduce bias.
  • Performance Variation: The performance of models varied significantly across different languages, indicating a need for more robust multilingual solutions.

Future work could focus on:

  • Improving Datasets: Creating larger and more balanced datasets, especially for code-mixed languages, would be beneficial.
  • Advanced Preprocessing: Exploring more sophisticated text preprocessing techniques could further enhance model performance.
  • Model Optimization: Investigating better embeddings and fine-tuning transformer models specifically for homophobia/transphobia detection in diverse linguistic contexts.
  • Language Agnosticism: Developing more effective language-agnostic models that can handle a wider range of languages and code-mixing scenarios.

Summary updated

BibTeX
@inproceedings{swaminathan2022ssncse,
  author = {Krithika Swaminathan and Bharathi B and Gayathri G L and Hrishik Sampath},
  title = {{SSNCSE\_NLP@LT-EDI-ACL2022: Homophobia/Transphobia Detection in Multiple Languages using SVM Classifiers and BERT-based Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure