TL;DR
This paper introduces a novel dynamic word embedding model that learns attribute-specific word representations, allowing for a more nuanced understanding of how language reflects societal biases. After correcting significant implementation errors in their original work, the authors demonstrate that their dynamic model can better capture subtle linguistic biases, particularly in sparser datasets like talk radio transcripts, compared to traditional static embedding approaches.
What Problem Does This Paper Address?
Word embeddings, which represent words as numerical vectors, have become powerful tools for uncovering hidden biases and stereotypes embedded within large text corpora. Previous work often trained separate embedding models for different contexts (e.g., different decades) and then aligned them to compare biases over time. This approach, however, is inefficient as it discards valuable data that could be pooled for better training and regularization. Furthermore, it doesn't fully leverage the potential of dynamic embeddings that can learn context-specific representations. This paper aims to address these limitations by proposing a unified dynamic embedding model that jointly learns general word semantics alongside attribute-specific offsets, enabling a more integrated and potentially more accurate analysis of linguistic biases.
Methods
The core of this paper is a dynamic word embedding model that extends the distributed memory (DM) model for learning paragraph vectors. The model learns two components for each word:
- Attribute-invariant embedding ($\gamma_w$): This represents the general meaning of a word across all contexts.
- Attribute-specific offset ($\beta_a$): This captures how the word's meaning shifts based on a specific attribute (e.g., year, day, city).
The final word embedding $E(w, A)$ is a sum of the invariant embedding and the relevant attribute-specific offset. This model is trained using a neural network with a negative sampling loss function, similar to Word2Vec. The authors implemented this model using PyTorch and incorporated L2 regularization specifically on the attribute-specific offsets to encourage them to capture only necessary contextual variations.
Datasets
The paper utilizes two primary datasets:
- Corpus of Historical American English (COHA): A large historical corpus used to investigate gender and ethnic occupation biases. This dataset was previously used in related work by Garg et al. (2018).
- Talk Radio Transcripts: A novel corpus of 119 million words from 83 stations across 64 US cities, collected over a one-month period. This dataset was used to analyze biases against refugees.
Key Insights and Ideas
- Correcting Errors is Crucial: The paper highlights the significant impact of seemingly small implementation errors (e.g., list concatenation instead of vector addition, incorrect indexing) on embedding model results. Their corrected model produced drastically different and more sensible outputs.
- Dynamic Embeddings for Sparsity: The dynamic model appears to be more suitable for representing linguistic biases in sparser datasets (like the talk radio data, where daily data is limited) compared to training individual static models for each attribute value.
- Attribute-Specific Offsets Matter: The model's ability to learn attribute-specific offsets is key to capturing nuanced shifts in word meaning, which can reveal subtle biases that static models might miss.
- Regularization Strategy: Applying L2 regularization primarily to attribute-specific offsets is a deliberate choice to ensure that general word semantics are learned robustly, while contextual variations are captured only when necessary.
Key Results
- COHA Analysis (Gender/Ethnic Bias): After correcting errors, the dynamic model's gender occupation bias scores aligned well with previous work and ground truth workforce participation rates. However, the ethnic occupation bias scores showed less alignment, suggesting potential challenges in capturing these specific biases with the current model.
- Talk Radio Analysis (Refugee Bias): The dynamic model produced more "smooth" bias scores that seemed to better track the news cycle and reflect a generally unwelcoming discourse towards refugees, aligning with the known leanings of conservative talk radio. In contrast, the non-dynamic model produced noisier scores and often negative bias values, suggesting it failed to capture the prevailing sentiment.
- Commonsense Word Association: Sanity checks using word association tests revealed that the original model produced nonsensical word similarities (e.g., "internet" appearing in 1910), whereas the corrected model yielded intuitive and historically accurate associations.
Limitations and Future Work
- Data Sparsity Trade-offs: While the dynamic model shows promise for sparse data, the paper acknowledges that when ample data is available for each attribute (like COHA by decade), traditional methods might still be competitive.
- Hyperparameter Tuning: The authors suggest exploring different values for the regularization hyperparameter ($\lambda$) to better understand its impact on attribute-specific embedding influence.
- Ground Truth Selection: Evaluating bias detection models remains challenging. Future work should focus on more thoughtful selections of "ground truth" for model validation.
- Comparison to Other Dynamic Models: The paper calls for comparisons with other dynamic embedding models that treat attributes as continuous variables.
- Broader Applicability: Further research is needed to explore the model's effectiveness on topics beyond those studied and across different time slices and discourse types.
- Geographic Bias Analysis: The speculative analysis on geographic bias was removed due to weaker model validation in the corrected version, indicating a need for more robust foundational results before exploring such complex attributes.
Summary updated