Can Language Models Guess Your Identity? Analyzing Demographic Biases in AI Essay Scoring

Alexander Kwako, Christopher OrmerodNAACL 2024

TL;DR

This paper investigates demographic biases in an AI essay scoring system (XLNet) trained on a large corpus of student essays. While the model performs well, it subtly magnifies existing human biases, favoring female and White students. Furthermore, the model's internal representations show some ability to predict demographic affiliations, suggesting these biases might be implicitly learned.

What Problem Does This Paper Address?

Automated Essay Scoring (AES) systems, particularly those leveraging Large Language Models (LLMs), are increasingly used in education. However, LLMs are trained on vast amounts of text data that can contain societal biases. If these biases are not carefully monitored and addressed, AES systems risk perpetuating and even amplifying existing inequities, potentially leading to unfair outcomes for vulnerable student populations. This paper aims to rigorously analyze an LLM-based AES system for demographic biases, specifically examining if the model's scoring reflects or exacerbates differences observed in human scoring, and whether demographic information is implicitly encoded within the model's internal workings.

Methods

The researchers fine-tuned an XLNet model, a transformer-based LLM known for handling long contexts, for automated essay scoring. They used a regression head with Mean Squared Error (MSE) loss, which is standard for retaining the ordinal nature of scores. To evaluate performance, they employed common metrics: Cohen's quadratic weighted kappa (QWK), standardized mean difference (SMD), and exact agreement.

To analyze demographic biases, they performed pairwise comparisons between different demographic groups (e.g., male vs. female, Black vs. White). To control for marginal differences in essay characteristics (like length) that might confound the analysis, they used an "exact matching" technique to create sample weights that effectively equalize these marginal differences. This allowed them to isolate the impact of demographic affiliation on scoring.

Additionally, they investigated whether demographic information was "embedded" within the XLNet scoring model. This was done by training linear models on the hidden states of the fine-tuned XLNet scoring model to predict demographic affiliations. This approach helps determine if the model's internal representations, used for scoring, contain information predictive of a student's demographic group.

Datasets

The primary dataset used in this study is the PERSUADE corpus. This corpus contains over 25,000 persuasive student essays from grades 6-12, along with holistic scores assigned by human raters and rich demographic metadata for the students. The researchers used a standard train-test split from a previous Kaggle competition involving this corpus and created a development set for model selection and hyperparameter tuning. No new benchmarks were introduced; the study leverages an existing, well-annotated dataset.

Key Insights and Ideas

  • Subtle Magnification of Bias: The study found that while the XLNet model achieved strong performance comparable to human raters, it subtly magnified existing marginal differences in human scores. This means the AI didn't just replicate human bias; it amplified it slightly.
  • Implicit Demographic Features: The fact that demographic affiliations could be predicted from the XLNet model's hidden states is a significant finding. It suggests that the model, even when trained solely for scoring, learns representations that are correlated with demographic identity, raising questions about how this information might influence scoring.
  • Importance of Fairness Audits: The research underscores the critical need for ongoing, rigorous fairness evaluations of AI systems in education, especially for high-stakes applications. Even small biases can have significant consequences for students.
  • Construct Relevance is Key: The paper touches upon the idea that understanding why demographic differences exist (i.e., whether they relate to actual writing proficiency or external factors) is crucial for developing effective debiasing strategies.

Key Results

  • Model Performance: The XLNet model demonstrated strong performance, with an overall quadratic weighted kappa (QWK) exceeding human-human reliability. However, three prompts showed high standardized mean differences (SMDs), indicating potential issues for those specific tasks.
  • Demographic Biases:
    • XLNet gave slightly higher scores to female students compared to male students (z = 0.06, p = .0012).
  • XLNet gave slightly lower scores to Black students compared to White students (z = -0.07, p = .0023).
  • These differences were consistent with marginal differences in human scores but were magnified by the model.
  • Students with lower SES and English Language Learner status also scored lower, but these differences were not statistically significant.
  • Predicting Demographics: When predicting demographic affiliation from the model's hidden states, the "kappa" (κ) values (an effect size measure) were generally low, ranging from "no agreement" to "minimal agreement." The exception was predicting ELL status for one prompt (κ = 0.75), indicating substantial predictive power for that specific group and prompt. When using text features alone to predict demographics, kappa values were higher, suggesting that the text itself contains more direct demographic signals than the scoring model's internal representations.

Limitations and Future Work

  • Human Rater Bias: The study's claims are limited to evaluating biases relative to human scores. Since human scores can also be biased, it's possible XLNet is actually fairer than human raters in some instances, despite magnifying marginal differences. The lack of "unbiased" anchor items (for Differential Item Functioning analysis) prevents a definitive conclusion on this.
  • Implicit vs. Explicit Use: While demographic affiliations were found to be embedded in the model's hidden states, the study could not definitively determine how this information was used (or if it was used at all) in generating the final essay scores.
  • Single Model and Dataset: The findings are based on one LLM (XLNet) and one dataset (PERSUADE). Future research should explore other state-of-the-art models and diverse essay datasets to assess the generalizability of these findings.
  • Future Directions: The authors suggest exploring additional LLMs and datasets, investigating the sources of group differences (e.g., language differences), and determining the "construct relevance" of these differences. They also propose exploring bias mitigation techniques during model training and scoring, such as orthogonal projection to remove demographic information from model representations, while acknowledging potential trade-offs with model performance.

Summary updated

BibTeX
@inproceedings{kwako2024can,
  author = {Alexander Kwako and Christopher Ormerod},
  title = {{Can Language Models Guess Your Identity? Analyzing Demographic Biases in AI Essay Scoring}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure