TL;DR
This paper investigates gender bias in Chinese word embeddings, specifically focusing on adjectives. By comparing automatically generated bias scores with human judgments, the authors reveal that while embeddings accurately capture stereotypes for adjectives associated with women, they tend to neutralize adjectives that people perceive as masculine, suggesting a divergence between learned representations and societal attitudes.
What Problem Does This Paper Address?
Word embeddings, dense vector representations of words, have become a cornerstone of Natural Language Processing (NLP). However, these embeddings often inadvertently learn and propagate societal biases present in the training data. While much research has focused on identifying and mitigating gender bias in English word embeddings, particularly concerning occupations, there's a significant gap in understanding this phenomenon in non-English languages and specifically for descriptive words like adjectives. This paper aims to fill this gap by examining gender bias in Chinese adjectives embedded in static word representations, exploring how these biases align with or diverge from human perceptions.
Methods
The researchers employed two main approaches:
- Word Embedding Training: They trained Chinese word embeddings using two models:
- CBOW (Continuous Bag-of-Words): A standard word embedding model.
- CWE (Character-Enhanced Word Embedding): A model that incorporates character-level information, which is particularly relevant for Chinese where character meanings can influence word semantics. This allows them to investigate the impact of character-level information on gender bias.
- Gender Bias Measurement: They adapted the method proposed by Bolukbasi et al. (2016) to calculate a gender bias score for each adjective. This score is derived from the projection of an adjective's vector onto the "he – she" direction. A positive score indicates a stronger association with males, and a negative score indicates a stronger association with females.
Datasets
- Training Data: A large corpus of Chinese news reports collected by Sogou Labs was used for training the word embeddings. The corpus details (size, tokens, unique words) are provided.
- Evaluation Data: The Adjectives List with Gendered Skewness and Sentiment (AGSS) dataset, developed by Zhu and Liu (2020), was used for evaluation. This dataset consists of gender preference scores for Chinese adjectives, collected from human annotators. The paper notes that AGSS itself exhibits a gender skew, with more adjectives leaning towards male associations.
Key Insights and Ideas
- Divergence in Bias: A striking finding is the difference in how word embeddings represent adjectives perceived as feminine versus masculine. Embeddings seem to align well with human perceptions for adjectives associated with women, but they tend to neutralize adjectives that people strongly associate with men.
- Character-Level Information's Role: The study explored the impact of character-enhanced word embeddings (CWE). While CWE showed slightly better correlation with human scores for adjectives perceived as feminine, it didn't necessarily improve the representation of masculine-leaning adjectives. In some cases, the standard CBOW model even showed a stronger correlation for feminine-leaning adjectives.
- Societal vs. Linguistic Evolution: The authors hypothesize that the observed divergence for masculine-leaning adjectives might be due to language evolution. Adjectives that were historically associated with men may be becoming more neutral in modern usage (as reflected in online news corpora), while human attitudes might lag behind, relying on older perceptions.
Key Results
- Overall Bias: Both CBOW and CWE models produced a majority of adjectives with positive gender bias scores (closer to male), mirroring the skew in the AGSS dataset. However, the average bias score for both models was negative, suggesting that the magnitude of negative bias scores might be larger than positive ones.
- Correlation with Human Scores: The Pearson correlation coefficient between word embedding bias scores and AGSS scores was moderate (around 0.489-0.503). This indicates a positive association but also a significant difference between learned embeddings and human judgment.
- Subgroup Analysis:
- For adjectives with AGSS scores < 3 (leaning feminine), the correlation was stronger (0.673 for CBOW, 0.628 for CWE), suggesting better alignment between embeddings and human perception for feminine-associated adjectives.
- For adjectives with AGSS scores > 3 (leaning masculine), the correlation was very weak (0.036 for CBOW, 0.020 for CWE), highlighting the neutralization effect in embeddings.
Limitations and Future Work
- Static Embeddings: The study used static word embeddings, which do not capture contextual nuances. Future work could explore contextualized embeddings (like BERT) to see if they offer a more dynamic representation of gender bias.
- Corpus Bias: The training data (online news reports) might have its own inherent biases that influence the embeddings.
- AGSS Dataset Limitations: While valuable, the AGSS dataset has its own skew and limitations in capturing the full spectrum of human attitudes.
- Language Evolution Hypothesis: The hypothesis about language evolution needs further investigation with diachronic corpora and more sophisticated analysis.
- Debiasing: The paper focuses on measurement and comparison, not debiasing. Future work could explore debiasing Chinese word embeddings for adjectives.
Summary updated