AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings

Anne Lauscher, Rafik Takieddin, Simone Paolo Ponzetto, Goran Glavas

TL;DR

This paper introduces AraWEAT, a framework for analyzing biases in Arabic word embeddings across various dimensions like model type, text source, and dialect. They found that implicit gender bias in Arabic news embeddings has been steadily increasing over time and that dialectal Arabic embeddings exhibit more bias than Standard Arabic ones.

What Problem Does This Paper Address?

Word embeddings, the numerical representations of words used in many Natural Language Processing (NLP) tasks, have been shown to encode human biases like sexism and racism. This is problematic because these biased embeddings can then propagate and even amplify these biases in downstream NLP applications, potentially reinforcing social injustices. While research has extensively studied these biases in English, there's a significant gap in understanding and quantifying them in other languages, particularly Arabic, which is spoken by hundreds of millions worldwide. This paper aims to fill this gap by providing a comprehensive analysis of biases in Arabic word embeddings.

Methods

The core contribution is the AraWEAT framework, an Arabic adaptation of the multilingual XWEAT framework. AraWEAT leverages existing bias tests like the Word Embedding Association Test (WEAT), Embedding Coherence Test (ECT), Bias Analogy Test (BAT), and Implicit Bias Test (K-Means++).

Key methodological aspects include:

  • Translation and Adaptation: Existing English WEAT test specifications were translated into Modern Standard Arabic (MSA) and manually verified by a native speaker. Crucially, they accounted for Arabic's grammatical gender by translating gender-neutral terms (like "engineer") into both masculine and feminine forms to avoid introducing artificial bias.
  • Multidimensional Analysis: The study systematically analyzes biases across several dimensions:
  • Embedding Models: Comparing SKIP-GRAM, CBOW, and FASTTEXT.
  • Text Sources: Investigating corpora from Wikipedia, news, and user-generated content (Twitter).
  • Dialects: Contrasting Egyptian Arabic with Modern Standard Arabic (MSA).
  • Time Periods: Conducting diachronic analyses on corpora from different years.
  • Vector Sizes and Preprocessing: Examining the impact of dimensionality and unigram/n-gram preprocessing.
  • Bias Quantification: Employing both explicit bias measures (like WEAT) and implicit bias measures (like K-Means++ accuracy) to get a more holistic view.

Datasets

The paper uses several datasets for training and evaluation:

  • Training Corpora:
    • Leipzig Corpora Collection (for training custom embeddings): This includes Arabic news corpora from various time periods (2007-2017) and different sizes (300k and 1M sentences).
  • Wikipedia and CommonCrawl (for pretrained FT embeddings).
  • Twitter (for pretrained AV embeddings).
  • Evaluation Benchmarks:
    • AraWEAT: This is the newly introduced benchmark, consisting of translated and adapted bias test specifications from English WEAT. It's designed to measure various types of biases in Arabic word embeddings.
  • SemEval 2017 Task 1 for Arabic: Used to evaluate the semantic quality of the trained embedding models.

Key Insights and Ideas

  • Dialect Matters: Biases appear to be more pronounced in embeddings trained on dialectal (Egyptian) Arabic compared to Modern Standard Arabic. This highlights the importance of considering linguistic variation when analyzing bias.
  • Temporal Trend in Gender Bias: Implicit gender bias in Arabic news corpora has shown a steady increase over the decade from 2007 to 2017. This is a concerning trend suggesting that societal biases might be becoming more embedded in language technologies over time.
  • Corpus Size and Bias: Similar to findings in other languages, larger corpora tend to lead to more pronounced biases. Smaller corpora showed fewer significant bias effects, suggesting that bias might be more diluted or less consistently represented in smaller datasets.
  • Predicting Corpus Bias: The study found that the biases in embeddings trained on a large, aggregated corpus can be reasonably predicted by averaging the biases from its temporally distinct sub-corpora. This suggests a way to approximate bias analysis without needing to train on massive, continuous datasets.
  • Implicit vs. Explicit Bias: While explicit gender biases (measured by WEAT) were not always significant across all tested embeddings, implicit biases (measured by K-Means++) were often present, indicating that biases can be subtle and not always directly apparent.

Key Results

  • Egyptian Arabic vs. MSA: Embeddings trained on Egyptian Arabic showed slightly more pronounced explicit and implicit bias than those trained on MSA.
  • Gender Bias Trend: For embeddings trained on Arabic news corpora (300k and 1M sentences), significant explicit gender biases were observed in specific years (2009, 2015, 2017), while implicit gender bias steadily rose from 2007 to 2017.
  • Corpus Aggregation: A Pearson correlation of 66% was found between biases of aggregated corpora (AVG) and whole corpus embeddings (CONC), suggesting that aggregating biases from sub-corpora is a reliable predictor.
  • User-Generated Content: Contrary to some expectations, embeddings built from user-generated content (Twitter) did not encode more bias on average than those trained on Wikipedia.

Limitations and Future Work

  • Limited Dialect Coverage: While Egyptian Arabic was analyzed, the study did not translate WEAT specifications to other Arabic dialects, acknowledging that dialectal differences might influence bias.
  • WEAT Limitations: The authors acknowledge that WEAT, while useful, has been shown to potentially overestimate bias (Ethayarajh et al., 2019), which is why they coupled it with other tests.
  • Future Work: The authors hope that AraWEAT will encourage further research into bias in Arabic language technology. Potential future directions could include:
  • Expanding the analysis to more Arabic dialects.
  • Developing debiasing methods specifically for Arabic word embeddings.
  • Investigating other types of biases beyond gender.
  • Analyzing the impact of these biases on downstream Arabic NLP tasks.

Summary updated

BibTeX
@inproceedings{lauscher2020araweat,
  author = {Anne Lauscher and Rafik Takieddin and Simone Paolo Ponzetto and Goran Glavas},
  title = {{AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings}},
  booktitle = {WANLP @ COLING},
  year = {2020},
}
Figure