HeteroCorpus: A Corpus for Heteronormative Language Detection

Juan Vásquez, Gemma Bel-Enguix, Scott Thomas Andersen, Sergio-Luis Ojeda-TruebaNAACL 2022

TL;DR

This paper introduces HeteroCorpus, the first annotated dataset specifically designed to detect heteronormative language in English. By creating this corpus and evaluating baseline classification models, the authors aim to address a critical gap in NLP, enabling the development of more inclusive language technologies that move beyond traditional gender binaries.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has made significant strides in detecting and mitigating gender bias in language models. However, a crucial aspect often overlooked is "heteronormativity" – the pervasive assumption that heterosexuality is the default and most normal form of sexual orientation and relationships. This assumption, deeply embedded in societal structures and language, can lead to the marginalization and erasure of LGBTQIA+ individuals and other non-normative identities. While gender bias research has focused on the male/female binary, heteronormativity extends this to encompass a broader spectrum of identities and relationships. The authors argue that NLP systems, trained on biased data, can perpetuate these heteronormative assumptions, leading to real-world adverse consequences. They identify a lack of resources for studying and detecting heteronormative language, which this paper aims to fill.

Methods

The core of this paper is the creation and annotation of the HeteroCorpus. The process involved:

  • Data Collection: Tweets were scraped from Twitter using keywords related to gendered terms (e.g., "man," "woman," "husband," "girl"). The initial collection yielded 26,183 tweets.
  • Filtering: A multi-stage filtering process was applied. First, tweets containing specific adjective-noun or noun-adjective combinations indicative of gendered language were prioritized. Second, tweets with minimal semantic content (e.g., just a single keyword) were removed. This resulted in a final dataset of 7,265 tweets.
  • Annotation: Six annotators, all students from a linguistics lab, were tasked with labeling tweets as either "Heteronormative" (1) or "Non-Heteronormative" (0). Tweets in other languages or unintelligible content were labeled as "2" and discarded. Each tweet was annotated three times to ensure reliability.
  • Agreement Measurement: Cohen's Kappa and Fleiss' Kappa were used to measure inter-annotator agreement, indicating a moderate level of agreement, suggesting the inherent complexity of annotating heteronormativity.
  • Baseline Classification: The authors established a baseline by training and evaluating two types of supervised classification algorithms:
  • Traditional ML: Support Vector Machines (SVM) and Logistic Regression (LR) using bag-of-words with TF-IDF features.
  • Deep Learning: Four fine-tuned BERT models (BERT-base-cased, BERT-base-uncased, BERT-large-cased, BERT-large-uncased).

Datasets

  • HeteroCorpus: This is the primary contribution of the paper. It's a newly introduced, human-annotated dataset of 7,265 English tweets collected from 2020-2022, specifically designed for the detection of heteronormative language.
  • Existing Datasets (for context): The paper references other datasets used for gender bias detection (e.g., those for sexism classification, occupational gender stereotypes, and gender-inclusive coreference resolution) to contextualize their work, but HeteroCorpus is novel in its focus on heteronormativity.

Key Insights and Ideas

  • Heteronormativity as a distinct linguistic challenge: The paper highlights that heteronormativity is more than just gender bias; it's a political and social system that privileges heterosexuality as the norm, impacting language in ways that go beyond the male/female binary.
  • Complexity of annotation: The moderate inter-annotator agreement (Fleiss' Kappa of 0.4036) suggests that identifying heteronormative language is a nuanced task, requiring careful consideration and potentially more sophisticated annotation guidelines or annotator training.
  • BERT's potential for heteronormativity detection: The results indicate that large BERT models can outperform traditional ML methods, suggesting that deep learning architectures are promising for capturing the subtle linguistic cues of heteronormativity.
  • The need for specialized corpora: The authors emphasize that existing NLP resources are insufficient for addressing heteronormativity, underscoring the importance of creating specialized datasets like HeteroCorpus.

Key Results

  • The final HeteroCorpus contains 7,265 tweets.
  • The distribution of labels was 5,284 non-heteronormative tweets (0) and 1,981 heteronormative tweets (1).
  • Classification Performance:
    • SVM: Accuracy 0.64, F1-score 0.55
    • Logistic Regression: Accuracy 0.67, F1-score 0.50
    • BERT-base-uncased: Accuracy 0.63, F1-score 0.59
    • BERT-base-cased: Accuracy 0.68, F1-score 0.62
    • BERT-large-uncased: Accuracy 0.71, F1-score 0.72
    • BERT-large-cased: Accuracy 0.72, F1-score 0.72
  • The BERT-large models achieved the highest accuracy and F1-scores, outperforming traditional ML methods.
  • The authors note that the overall low results indicate the difficulty of the task and the need for further research.

Limitations and Future Work

  • Limitations:
    • Annotation Complexity: The moderate inter-annotator agreement suggests that heteronormativity is a complex concept to label consistently.
  • Potential Bias in Keywords: The initial keyword selection for data collection might introduce bias.
  • Limited Scope: The corpus is currently limited to English tweets.
  • Twitter Data: Reliance on Twitter data means the corpus reflects the specific language and discourse patterns of that platform.
  • Future Work:
    • Corpus Expansion: Expand the corpus with a larger set of tweets to capture more nuanced forms of heteronormative discourse.
  • Multi-Class Definition: Incorporate a more nuanced, multi-class definition of heteronormativity, moving beyond a simple binary.
  • Cross-Lingual Corpora: Create similar corpora for other languages, as heteronormativity is a global issue.
  • Advanced Detection/Mitigation Tools: Encourage the development of more sophisticated tools for detecting and mitigating heteronormative language in NLP systems.

Summary updated

BibTeX
@inproceedings{vasquez2022heterocorpus,
  author = {Juan Vásquez and Gemma Bel-Enguix and Scott Thomas Andersen and Sergio-Luis Ojeda-Trueba},
  title = {{HeteroCorpus: A Corpus for Heteronormative Language Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure