Analyzing Bias in Large Language Model Solutions for Assisted Writing Feedback Tools: Lessons from the Feedback Prize Competition Series

Perpetual Baffour, Tor Saxberg, Scott CrossleyACL 2023

TL;DR

This paper investigates bias in winning Large Language Model (LLM) solutions from the Feedback Prize competition, which aimed to improve automated writing feedback tools. While the first competition showed minimal bias in identifying discourse elements, the second competition, focused on predicting discourse effectiveness, revealed moderate bias, particularly against students from marginalized demographic groups.

What Problem Does This Paper Address?

Automated Writing Feedback Tools (AWFTs) hold significant promise for enhancing student learning by providing immediate, sophisticated feedback. Recent advancements in LLMs have dramatically increased AWFT capabilities. The Feedback Prize competition series aimed to accelerate AWFT development by crowdsourcing LLM solutions. However, these powerful models need rigorous evaluation, especially for potential biases that could disproportionately affect certain student populations, particularly those historically marginalized. This study addresses the critical need to assess the performance accuracy of these winning LLM solutions across various demographic factors like race/ethnicity, economic disadvantage, and English Language Learner (ELL) status.

Methods

The study analyzes winning solutions from two Feedback Prize competitions.

  • Feedback Prize 1.0: Focused on segmenting essays into discourse elements (e.g., lead, claim, evidence) and assigning labels. Performance was evaluated using word overlap and discourse label matching, with a macro F1 score as the primary metric. The analysis specifically examined the second, third, and sixth-place models.
  • Feedback Prize 2.0: Focused on predicting the effectiveness of discourse elements (Ineffective, Adequate, Effective). Models submitted probabilities for each effectiveness rating, and evaluation used a multi-class logarithmic loss. The study analyzed winning models from the "Efficiency Track," which prioritized computational efficiency. A secondary analysis involved recoding effectiveness into a binary "Effective" vs. "Non-Effective" label to mitigate issues with imbalanced data.

Bias analysis involved disaggregating performance accuracy (True Positive Rate for Prize 1.0, Likelihood Scores for Prize 2.0) by demographic factors using t-tests and ANOVAs.

Datasets

  • PERSUADE Corpus: A collection of approximately 25,000 argumentative essays written by U.S. students in grades 6-12. Essays were annotated for discourse elements and their effectiveness.
  • Feedback Prize 1.0: Utilized the full PERSUADE corpus.
  • Feedback Prize 2.0: Used a subset of the PERSUADE corpus (~6,900 essays) to create a more balanced distribution of effectiveness ratings (18% Ineffective, 24% Effective, 58% Adequate), compared to the full corpus (4% Ineffective, 80% Adequate, 16% Effective).

Key Insights and Ideas

  • Competition Success, but Limited Dissemination: The Feedback Prize competitions successfully crowdsourced numerous LLM solutions, but these algorithms were largely confined to the Kaggle platform, hindering their adoption in educational applications.
  • Bias Varies by Task: The nature of the task significantly impacts bias. Identifying discourse elements (Prize 1.0) showed minimal demographic bias, whereas predicting discourse effectiveness (Prize 2.0) exhibited moderate bias.
  • LLMs Amplify Data Bias: Even without explicit demographic training, LLMs can amplify existing biases present in the training data. This is particularly evident in Prize 2.0, where the model's predictions mirrored the disproportionate representation of "effective" writing from advantaged student groups in the human-annotated dataset.
  • Data Imbalance is a Major Factor: The imbalanced distribution of effectiveness labels in the PERSUADE corpus (especially the low number of "Ineffective" labels) played a crucial role in the observed bias in Prize 2.0. Recoding to a binary "Effective/Non-Effective" label helped mitigate some bias but did not eliminate it.
  • Importance of Pre-Adoption Assessment: The findings underscore the critical need to assess AWFT algorithms for bias before widespread deployment, especially when serving diverse student populations.

Key Results

  • Feedback Prize 1.0: Winning models demonstrated minimal bias across race/ethnicity, ELL status, and economic disadvantage. For instance, the difference in True Positive Rate (TPR) between ELL and non-ELL students was statistically significant (p=0.03) but had a very small effect size (Cohen's d=0.103).
  • Feedback Prize 2.0:
    • Original Effectiveness Ratings: Showed higher levels of bias. The first-place model was significantly more accurate in identifying effective discourses for non-ELL students (76% vs. 27% accurate) with a large effect size (Cohen's d ~0.671). Similar significant differences were observed for economically disadvantaged students (moderate effect size, Cohen's d ~0.263) and across race/ethnicity groups (small effect sizes, Cohen's d ~0.15).
  • Binary Effectiveness Ratings: While recoding to "Effective" vs. "Non-Effective" improved balance for the "Non-Effective" category, bias persisted in predicting "Effective" discourses, with white, non-ELL, and advantaged students still being overrepresented.

Limitations and Future Work

  • Data Bias: The study acknowledges that bias is likely inherent in any dataset that accurately reflects achievement disparities in the U.S. educational system.
  • Future Work: The authors suggest future research should focus on:
  • Identifying potential data biases in label distribution, agreement, and demographic representation before model development.
  • Developing strategies to mitigate bias in algorithmic outputs to ensure fairness in AWFT systems.
  • Exploring how to address current writing achievement disparities in the U.S. educational system, as these disparities contribute to data bias.

Summary updated

BibTeX
@inproceedings{baffour2023analyzing,
  author = {Perpetual Baffour and Tor Saxberg and Scott Crossley},
  title = {{Analyzing Bias in Large Language Model Solutions for Assisted Writing Feedback Tools: Lessons from the Feedback Prize Competition Series}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure