HOMO-MEX: A Mexican Spanish Annotated Corpus for LGBT+phobia Detection on Twitter

Juan Vásquez, Scott Andersen, Gemma Bel-enguix, Helena Gómez-adorno, Sergio-luis Ojeda-truebaACL 2023

TL;DR

This paper introduces HOMO-MEX, the first annotated corpus for detecting LGBT+phobia specifically in Mexican Spanish tweets. By creating this resource and evaluating various machine learning and deep learning models, the authors aim to advance NLP capabilities in understanding and combating online hate speech in a less-resourced linguistic variant.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has made significant strides in detecting hate speech, particularly in English. However, a critical gap exists in resources for other languages, especially for nuanced forms of discrimination like LGBT+phobia in Spanish. Existing Spanish corpora often focus on Peninsular Spanish, which differs lexically and pragmatically from Mexican Spanish. This linguistic variation means that models trained on one variant may not perform well on another, leaving a substantial portion of online discourse under-analyzed for LGBT+phobic content. The authors address this by creating a dedicated Mexican Spanish corpus and establishing a benchmark for its detection.

Methods

The authors employed a multi-stage approach:

  • Data Collection: They scraped public tweets from Twitter using an API, focusing on keywords indicative of the LGBT+ community in Mexican Spanish. This involved linguistic students identifying relevant nouns and their variations, including appreciative and gender-inflected forms.
  • Annotation Process: A custom annotation platform was used. Annotators, a mix of LGBT+ individuals and heterosexual individuals, were tasked with two main objectives:
  • Phobia Detection: Classifying tweets as "LGBT+phobic," "Not LGBT+Phobic," or "Irrelevant to the LGBT+ community."
    1. Fine-grained Classification: For tweets identified as LGBT+phobic, further categorizing them into "gayphobia," "lesbophobia," "biphobia," "transphobia," or "other LGBT+phobic content."
    2. To ensure robustness, tweets were annotated by four individuals, with a tie-breaking mechanism involving additional annotators and a final decision by a specialized annotator.
  • Classification Experiments: The researchers evaluated two main approaches:
  • Traditional Machine Learning: Naive Bayes, SVM, Logistic Regression, and Random Forest were trained using bag-of-words and TF-IDF features.
  • Pre-trained Deep Learning Models: Various BERT-based models (bert-base-multilingual-cased/uncased, beto-cased/uncased) were fine-tuned for classification tasks without extensive pre-processing.

Datasets

The primary contribution is the HOMO-MEX corpus, a newly introduced benchmark for LGBT+phobia detection in Mexican Spanish.

  • Size: The corpus consists of 11,000 annotated tweets.
  • Subsets: It is divided into two main subsets:
  • LGBT+Phobia Detection Subset: For binary classification (LGBT+phobic vs. not/irrelevant).
  • Fine-grained Classification Subset: For multi-label classification of specific phobia types.
  • Data Source: Publicly available tweets scraped via Twitter's API, filtered by Mexican geolocation.

Key Insights and Ideas

  • Linguistic Nuance Matters: The study highlights that Mexican Spanish has unique linguistic features and pragmatic uses that necessitate language-specific corpora for accurate hate speech detection.
  • Annotator Agreement Challenges: Achieving high inter-annotator agreement, especially for fine-grained categories, proved difficult. This is attributed to the complexity of differentiating irony, re-signification, and humor within the LGBT+ community, which can be challenging even for insiders.
  • Gendered Disparities in Phobia: Surprisingly, the authors observed more hateful speech when referring to the masculine gay community compared to feminine terms. This suggests that while the corpus aims to capture LGBT+phobia, the invisibility of women within broader discussions of sexism might be a related, yet distinct, issue.
  • Benefit of the Doubt: The annotation guidelines encouraged annotators to give the "benefit of the doubt" to authors, aiming to avoid over-labeling ironic or self-deprecating LGBT+phobic content that doesn't represent true harm.

Key Results

  • Traditional ML Performance: SVM generally performed best across most metrics for the LGBT+Phobia detection subset.
  • BERT Performance: Beto-cased achieved the highest classification metrics for the LGBT+Phobia detection subset, while bert-base-multilingual-uncased outperformed other BERT models on the fine-grained classification subset.
  • Accuracy Metrics:
    • For the LGBT+Phobia detection subset, SVM achieved an F1-score of 0.7670.
  • For the fine-grained classification subset, SVM achieved an F1-score of 0.6909.
  • BERT models showed accuracies in the range of 0.76-0.86 for LGBT+Phobia detection and 0.73-0.78 for fine-grained classification.
  • Annotator Agreement: Fleiss' Kappa scores indicated moderate agreement for the overall phobia detection task but lower consistency for fine-grained categories, with non-LGBT+ and male annotators showing higher agreement in some areas.

Limitations and Future Work

  • Annotation Agreement: The authors acknowledge the challenge of annotator agreement due to the difficulty in distinguishing irony, re-signification, and humor within the LGBT+ community.
  • Internalized Stereotypes: Potential influence of internalized stereotypes on annotator behavior is noted.
  • "Other" Category: Low agreement for the "Other" category suggests it may be too broad or that nuanced LGBT+phobia types are hard to identify.
  • Future Work:
    • Expand the dataset with more diverse terms to represent the full spectrum of the LGBT+ community.
  • Develop a more profound labeling system to reduce ambiguity and incorporate non-binary labels.
  • Explore categories like derogatory, threatening, humor, and neutral comments.
  • Create more representative datasets for other Spanish variants to enable dialectal approaches.
  • Incorporate lexicon-informed approaches for context-aware features in classification models.

Summary updated

BibTeX
@inproceedings{vasquez2023homo,
  author = {Juan Vásquez and Scott Andersen and Gemma Bel-enguix and Helena Gómez-adorno and Sergio-luis Ojeda-trueba},
  title = {{HOMO-MEX: A Mexican Spanish Annotated Corpus for LGBT+phobia Detection on Twitter}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure