AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings

Anne Lauscher, Rafik Takieddin, Simone Paolo Ponzetto, Goran Glavas

Introduction

Word embeddings serve as the foundational vector representations for modern Natural Language Processing (NLP). However, because these models are trained on vast corpora of human-generated text, they frequently inherit and amplify societal prejudices. While "representational harm" is well-documented in English-centric models, Arabic presents unique linguistic challenges. Its high morphological richness, grammatical gender for nearly all nouns, and distinct regional dialects (e.g., Modern Standard Arabic vs. Egyptian) complicate the measurement of bias.

To address this, the authors introduce AraWEAT, a framework designed to quantify "stereotypical unfairness" across various Arabic models, dialects, and temporal snapshots.

The AraWEAT Framework

AraWEAT extends the multilingual XWEAT framework by integrating four distinct bias metrics to provide a holistic view of model behavior:

  1. Word Embedding Association Test (WEAT): Measures explicit associations between target sets (e.g., Male/Female) and attribute sets (e.g., Career/Family) via cosine similarity.
  2. Embedding Coherence Test (ECT): Evaluates the consistency of distances between target terms and attribute terms.
  3. Bias Analogy Test (BAT): Quantifies biased analogy completions (e.g., "Man is to Doctor as Woman is to X").
  4. K-Means++ (KM): An implicit bias test that assesses how easily an algorithm can cluster target terms based solely on their vectors.
Overview of bias tests and their corresponding target and attribute sets.

A critical contribution of AraWEAT is its gender-aware translation. Unlike English, Arabic is not gender-neutral; therefore, the researchers manually ensured that neutral terms were translated into both masculine and feminine forms (e.g., muhandis and muhandisa for "engineer") to prevent the introduction of artificial bias during testing.

Datasets and Evaluation

The study evaluated several prominent Arabic embedding spaces, including FastText (Wikipedia/CommonCrawl) and AraVec (Wikipedia/Twitter), covering both Modern Standard Arabic (MSA) and Egyptian dialects. Additionally, the researchers performed a diachronic analysis using the Leipzig News Corpora, examining yearly snapshots from 2007 to 2017.

Key Insights and Results

The analysis revealed several significant trends regarding how bias manifests in Arabic NLP:

  • Dialectal Variance: Bias was notably more pronounced in Egyptian Arabic embeddings compared to MSA. For instance, in FastText models, the WEAT score for gender bias (Test 7) was 0.65 for Egyptian but a statistically insignificant -0.15 for MSA. This suggests that colloquial data may capture more "raw" societal stereotypes than formal, edited text.
  • Temporal Trends in News: Implicit gender bias in news corpora (measured via KM) showed a steady increase over a decade, rising from 0.53 in 2008 to 0.72 in 2017. This indicates that the digital news landscape has become increasingly polarized or stereotypical in its gender-related discourse.
  • Corpus Size and Signal: Explicit bias was only reliably detectable in larger corpora (over 1 million sentences). In smaller datasets, the bias signal is often obscured by the noise of lower-quality vectors.
Bias scores across different Arabic models and dialects for various tests.

Interestingly, embeddings derived from user-generated content (Twitter) did not consistently show higher bias than those from curated sources like Wikipedia, challenging the assumption that social media is inherently more biased for training data.

Limitations and Future Work

While AraWEAT provides a robust starting point, the authors note several limitations. The study excluded proper names due to cultural translation difficulties and focused primarily on MSA and Egyptian dialects, leaving Levantine, Maghrebi, and Gulf dialects for future exploration. Furthermore, the tests were limited to standard categories like gender and career; Arabic-specific social biases, such as tribalism or religious sectarianism, were not included.

Future research should utilize AraWEAT to evaluate the efficacy of debiasing algorithms, particularly in the context of morphologically complex languages where standard English-centric approaches may fail.

Summary updated

Abstract

Recent work has shown that distributional word vector spaces often encode human biases like sexism or racism. In this work, we conduct an extensive analysis of biases in Arabic word embeddings by applying a range of recently introduced bias tests on a variety of embedding spaces induced from corpora in Arabic. We measure the presence of biases across several dimensions, namely: embedding models (Skip-Gram, CBOW, and FastText) and vector sizes, types of text (encyclopedic text, and news vs. user-generated content), dialects (Egyptian Arabic vs. Modern Standard Arabic), and time (diachronic analyses over corpora from different time periods). Our analysis yields several interesting findings, e.g., that implicit gender bias in embeddings trained on Arabic news corpora steadily increases over time (between 2007 and 2017). We make the Arabic bias specifications (AraWEAT) publicly available.
BibTeX
@inproceedings{lauscher2020araweat,
  author = {Anne Lauscher and Rafik Takieddin and Simone Paolo Ponzetto and Goran Glavas},
  title = {{AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings}},
  booktitle = {WANLP @ COLING},
  year = {2020},
}
Figure