Bad Seeds: Evaluating Lexical Methods for Bias Measurement

Maria Antoniak, David MimnoACL 2021

TL;DR

This paper investigates the critical role of "seed sets" – lists of words used to measure bias in language models – and reveals how their selection and characteristics significantly impact measurement results. The authors highlight the need for greater transparency and rigorous evaluation of these seeds to ensure reliable and meaningful bias detection in NLP.

What Problem Does This Paper Address?

As natural language processing (NLP) models become more pervasive, understanding and mitigating the biases they encode is crucial. A common approach to measuring these biases involves using "seed sets," which are curated lists of words designed to represent specific concepts or stereotypes (e.g., words associated with "gender" or "race"). However, there's a lack of clear guidance on how these seed sets are chosen, and their potential influence on measurement outcomes is often overlooked. This can lead to unreliable bias assessments, making it difficult to compare results across different studies or datasets, and potentially obscuring or even amplifying existing biases. The paper aims to address this by systematically analyzing the sources, characteristics, and impact of these seed sets on bias measurement.

Methods

The authors employ a multi-faceted approach:

  • Literature Survey and Documentation: They gathered and documented 178 seed sets from prior bias measurement research, categorizing their sources (e.g., borrowed from social sciences, crowd-sourced, corpus-derived).
  • Qualitative Analysis: They performed a close reading of example seed terms to identify potential issues like reductive definitions or embedded stereotypes.
  • Quantitative Analysis: They used established bias measurement algorithms like the Word Embedding Association Test (WEAT) and Principal Component Analysis (PCA) to quantify how variations in seed sets affect measurement results. This involved training word embeddings on various corpora and then testing different seed sets on these embeddings.
  • Generated Seed Sets: To isolate the impact of specific seed features (like frequency and POS), they generated artificial seed sets.

Datasets

The paper uses several datasets for training word embeddings and evaluating bias measurement methods:

  • New York Times (NYT): A collection of articles from April-June 2016, chosen for its cultural specificity and manageable size.
  • WikiText: A large dataset of manually verified Wikipedia articles, offering a broader scope.
  • Goodreads: Samples of book reviews from the romance and history/biography genres, selected for their cultural specificity.
  • Google News: A pre-trained model on a large Google News corpus was used for comparison with existing studies.

The paper notes that these datasets are generally smaller and more specialized than those typically used for large-scale NLP applications, fitting the use case of social scientists measuring bias in specific contexts.

Key Insights and Ideas

  • Seeds are not neutral: Seed sets themselves can contain cultural and cognitive biases, and their linguistic features (like frequency) can significantly skew measurement results.
  • Re-use without validation is risky: Many seed sets are re-used across studies without sufficient validation for new contexts, leading to potential mismatches in meaning and frequency.
  • Source matters: Different sources for seed sets (e.g., social science literature vs. crowd-sourcing) come with their own set of risks and benefits that researchers must consider.
  • Instability is widespread: Even for well-established bias dimensions like gender, different seed sets can lead to vastly different conclusions about the presence and magnitude of bias.
  • Fine-grained analysis is crucial: The paper demonstrates that subtle changes, like shuffling the order of seed pairs or using similar seed sets, can dramatically alter bias measurement outcomes.

Key Results

The study found that:

  • Seed choice significantly impacts bias measurements: Figure 2 shows that different female seed sets can lead to widely varying similarity scores to an "unpleasantness" vector, even within the same corpus.
  • Shuffling seed pairs can alter bias subspaces: Figure 3 illustrates how shuffling gender pairs can change the dominant component in PCA analysis, suggesting that the order and pairing of seeds are important.
  • Set similarity correlates with explained variance: Figure 5 indicates that more similar seed sets tend to explain less variance in bias measurements, highlighting the importance of diverse and distinct seed terms.
  • Coherence scores vary widely: Table 4 shows that generated seed sets can achieve higher coherence (indicating greater separation in the bias subspace) than gathered seed sets, suggesting that carefully constructed or even artificial sets can sometimes be more discriminative.

Limitations and Future Work

The authors acknowledge several limitations:

  • Unigram focus: The analysis primarily focuses on unigram seeds, excluding bigram seeds for simplicity.
  • Attribute term simplification: In WEAT experiments, attribute terms were held constant (e.g., "good" vs. "bad") to focus on seed variation, which is a simplification of real-world scenarios.
  • Dataset limitations: While chosen for specific reasons, the datasets are not exhaustive and may not fully represent all potential contexts.

Future work suggested includes:

  • Extending to contextual embeddings: While the study focused on vocabulary-based embeddings for easier analysis, extending findings to contextual embeddings is important.
  • Developing better documentation standards: The paper strongly advocates for more consistent and transparent documentation of seed sets and their rationales.
  • Further exploration of seed features: Investigating the impact of other seed features like POS tags and semantic similarity in more depth.
  • Creating standardized evaluation protocols: The authors implicitly call for more rigorous and standardized methods for evaluating seed sets before they are used in bias measurement.

Summary updated

BibTeX
@inproceedings{antoniak2021bad,
  author = {Maria Antoniak and David Mimno},
  title = {{Bad Seeds: Evaluating Lexical Methods for Bias Measurement}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure