Wanted: Personalised Bias Warnings for Gender Bias in Language Models

Chiara Di Bonaventura, Michelle Nwachukwu, Maria StoicaACL 2025

TL;DR

This paper addresses the critical gap in how gender bias in large language models (LLMs) is communicated to users. Through a user study with female AI practitioners, they found that while there's consensus on when and how to mitigate and measure bias, user preferences for how bias is communicated vary significantly. They propose a novel, customizable framework for personalized bias warnings that leverages runtime monitoring.

What Problem Does This Paper Address?

Large Language Models (LLMs) are becoming ubiquitous, but they often inherit and amplify societal biases, particularly gender bias. While significant research has focused on detecting and mitigating these biases, there's a notable lack of focus on how to effectively communicate these biases to the end-users, the practitioners who deploy these models. This communication gap can hinder user trust and understanding of LLM limitations. The authors argue that a "one-size-fits-all" approach to bias warnings is insufficient and that personalized communication is key.

Methods

The core of this research involved a user study conducted at an ACM WomENcourage workshop. The study targeted female AI practitioners and included:

  • Bias Mitigation Presentation: Discussing how bias enters LLM pipelines (data collection, model development, evaluation).
  • Bias Measurement Presentation: Explaining intrinsic (e.g., word embeddings) and extrinsic (e.g., downstream task performance) metrics.
  • Pilot Study: A hands-on session with three open-ended questions designed to gather user perspectives on:
  • The most critical stage for bias intervention in the LLM pipeline.
  • The effectiveness of intrinsic vs. extrinsic bias measurement.
  • Preferred methods for receiving bias warnings.

Based on the study's findings, the authors propose a personalized bias warning framework. This framework leverages runtime monitoring techniques, similar to those used for detecting misclassifications or out-of-distribution data. The monitor analyzes model inputs, outputs, and previous monitor outputs to generate quantitative bias scores. These scores can then be visualized and customized, offering a flexible approach to bias communication.

Datasets

The paper does not explicitly mention specific datasets used for training or evaluating LLMs within the scope of this particular study. The focus is on user preferences and a proposed framework, rather than empirical evaluation on benchmark datasets.

Key Insights and Ideas

  • Consensus on Mitigation & Measurement, Variation in Communication: Practitioners largely agree on the importance of data curation for bias mitigation and that both intrinsic and extrinsic metrics are valuable for measurement. However, there's a wide divergence in how they prefer to be warned about bias.
  • The Need for Personalization: The study strongly suggests that a one-size-fits-all approach to bias warnings is inadequate. Users have diverse needs and technical backgrounds, necessitating tailored communication.
  • Runtime Monitoring for Dynamic Bias Insights: The proposed framework's reliance on runtime monitoring allows for continuous, input-by-input bias assessment. This is crucial because bias can manifest dynamically and contextually.
  • Bridging the Gap Between Technical Metrics and User Understanding: Visualizations and benchmark scores emerged as preferred warning methods, indicating a desire for clear, intuitive ways to grasp bias without necessarily diving deep into complex metrics.
  • Data Curation is Paramount: The overwhelming consensus among participants was that data curation is the most critical step for mitigating gender bias in LLMs, highlighting the "garbage in, garbage out" principle.

Key Results

  • Bias Mitigation: 19 out of 19 participants identified "Data curation" as the most important step for bias mitigation. "Evaluation" was mentioned by 4 participants, and "Development" by 1.
  • Bias Measurement: 63% of participants preferred using "Both" intrinsic and extrinsic metrics, while 32% favored "Extrinsic" measures. Only 1 participant preferred "Intrinsic."
  • Bias Warnings: The most preferred warning types were "Visualisation" (8 participants) and "Benchmark scores" (7 participants). "Explanation" was favored by 3, "Caution alert" by 2, and "Data distribution" by 2. "Argumentation" was chosen by 1 participant.
  • Qualitative Findings: Participants emphasized that bias is context-dependent and that cultural/historical patterns in data can be both problematic and, in some applications, helpful. They also highlighted the importance of understanding how input variations affect bias.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Sample Size: The user study was a pilot with 19 participants, necessitating a larger study for more statistically significant results.
  • Demographic Homogeneity: The study primarily included female participants due to the workshop's focus. Future work should include a more diverse set of respondents regarding gender, race, and experience levels.
  • Binary Gender Focus: The study treated gender as binary (male/female), excluding non-binary and transgender individuals. Future work should explore these perspectives.
  • Domain Specificity: The study focused on general LLMs but participants noted that preferences might differ across domains (e.g., medical, machine translation).
  • Framework Evaluation: The proposed framework was developed based on pilot study findings, but participants were not asked to evaluate it directly.

Future work should involve evaluating the proposed framework on established datasets, conducting before-and-after user studies to assess its impact on user trust, and exploring its applicability in different domains and use cases. Collecting user feedback on the framework itself is also a key next step.

Summary updated

BibTeX
@inproceedings{bonaventura2025wanted,
  author = {Chiara Di Bonaventura and Michelle Nwachukwu and Maria Stoica},
  title = {{Wanted: Personalised Bias Warnings for Gender Bias in Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure