Introduction
Large language models (LMs) often act as mirrors of the internet, reflecting both its vast knowledge and its deep-seated societal prejudices. While significant research has focused on measuring bias in general-purpose models like BERT or GPT-2, conversational AI has remained relatively under-explored. This gap is critical because chatbots interact directly with users in dynamic, unpredictable ways.
The researchers identify two primary challenges in this field: the lack of realistic data and the potential performance trade-offs associated with debiasing. Most existing benchmarks rely on artificially constructed sentences that do not resemble real human dialogue. Furthermore, there is a concern that debiasing might act as a "linguistic lobotomy," degrading a model's ability to generate coherent responses or track conversation states. This paper introduces REDDITBIAS, a framework designed to evaluate and mitigate bias in conversational LMs using real-world human interactions.
The REDDITBIAS Dataset
The core contribution is the REDDITBIAS dataset, which consists of real Reddit comments manually annotated for bias across four dimensions: gender, race, religion, and queerness. Unlike previous "toy" datasets, these are actual human conversations, capturing the nuance and complexity of online discourse.
To evaluate the impact of debiasing on task performance, the authors also utilized MultiWoZ 2.0 for Dialogue State Tracking (DST) and DSTC-7 for Conversational Response Generation (CRG), alongside a Reddit 6K reference set for measuring in-domain perplexity.
Methodology
The researchers benchmarked DialoGPT using four distinct debiasing strategies adapted from general LM literature:
- Language Model Debiasing (LMD): An auxiliary loss function penalizing the model for assigning different probabilities to paired terms (e.g., "man" vs. "woman") in identical contexts.
- Attribute Distance Debiasing (ADD): A loss function that ensures internal representations of different groups (e.g., Christians vs. Muslims) are equally distant from stereotypical negative attributes.
- Hard Debiasing (HD): A mathematical approach that identifies a "bias subspace" in embeddings and neutralizes attribute words by making them orthogonal to that direction.
- Counterfactual Data Augmentation (CDA): A data-centric method where the model is fine-tuned on versions of the training data with group-identifying terms swapped.
Bias was measured using Language Model Bias (LMB), a perplexity-based metric. A model is considered biased if it finds a stereotypical sentence (e.g., "Jews are greedy") significantly more "natural" (lower perplexity) than a neutral alternative.
Key Results and Analysis
The study revealed that DialoGPT, despite being trained on data filtered for offensive phrases, still harbored significant latent religious biases against Jews and Muslims. This suggests that simple blacklists are insufficient for addressing structural bias.
Bias Mitigation vs. Perplexity
As shown in the results, Hard Debiasing (HD) and Counterfactual Augmentation (CDA) were the most effective methods, successfully removing religious bias to a statistically significant degree. However, the researchers noted an "over-correction" risk: in dimensions where the model was not initially biased (such as race), some techniques pushed the model into "anti-stereotypical" territory, highlighting the need for surgical rather than broad-brush debiasing.
Interestingly, while some methods like LMD drastically increased the model's perplexity (making it "confused" by standard text), the model's ability to perform downstream tasks remained stable. DialoGPT’s DST F1 score of .9444 was virtually unchanged after debiasing, suggesting that conversational performance is robust even when intrinsic LM metrics fluctuate.
Conclusion and Future Work
The REDDITBIAS framework demonstrates that it is possible to make chatbots more ethical without sacrificing their utility. However, the authors acknowledge limitations, including the binary treatment of gender and the lack of intersectional analysis (e.g., the combined effect of race and gender bias). Future work will focus on these complex, intersecting dimensions and the development of even more sophisticated real-world conversational datasets.
Summary updated