BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs

Zhiting Fan, Ruizhe Chen, Ruiling Xu, Zuozhu Liu

Abstract

Evaluating the bias in Large Language Models (LLMs) becomes increasingly crucial with their rapid development. However, existing evaluation methods rely on fixed-form outputs and cannot adapt to the flexible open-text generation scenarios of LLMs (e.g., sentence completion and question answering). To address this, we introduce BiasAlert, a plug-and-play tool designed to detect social bias in open-text generations of LLMs. BiasAlert integrates external human knowledge with inherent reasoning capabilities to detect bias reliably. Extensive experiments demonstrate that BiasAlert significantly outperforms existing state-of-the-art methods like GPT4-as-A-Judge in detecting bias. Furthermore, through application studies, we demonstrate the utility of BiasAlert in reliable LLM bias evaluation and bias mitigation across various scenarios. Model and code will be publicly released.

Introduction

As Large Language Models (LLMs) transition from research environments to real-world applications, the social biases they inherit from their training data pose significant risks. Detecting these biases is critical for safety and fairness. However, traditional evaluation methods are largely "fixed-form," relying on token probabilities or multiple-choice selections. These methods are ill-suited for the "open-text" reality of modern AI, where models generate free-form sentences, stories, or complex answers.

Furthermore, LLMs often lack the internal "social-ethical knowledge" required to judge their own outputs objectively. To bridge this gap, we present BiasAlert, a plug-and-play tool designed to audit any LLM's output in real-time. BiasAlert operates without requiring access to a model's internal weights or probabilities, making it a versatile supervisor for diverse AI architectures.

The BiasAlert Framework

BiasAlert treats bias detection as a retrieval-augmented reasoning task. Its architecture is built upon three primary pillars:

  1. Social Bias Retrieval Database: The framework utilizes a curated database of approximately 41,000 entries across seven categories (including race, gender, and religion) derived from the Social Bias Inference Corpus (SBIC).
  2. Retrieval-Augmented Input: When an LLM generates text, BiasAlert employs the Contriever model to retrieve the top-K most relevant examples of known human social knowledge from its database. This provides the external "moral compass" necessary for grounded judgment.
  3. Instruction-Following Reasoning: BiasAlert utilizes a fine-tuned Llama-2-7b-chat model to process the generated text alongside the retrieved references. The model follows a structured reasoning process: identifying target demographic groups, reviewing retrieved context, and determining the presence of bias with a detailed explanation.

Experimental Setup

To evaluate BiasAlert, we utilized several diverse datasets:

  • SBIC: Used for building the retrieval database and training the instruction-following component.
  • RedditBias: Real-world conversational data used for performance benchmarking.
  • Crows-pairs: A benchmark for measuring stereotypes.
  • BOLD & BeaverTails: Used to test BiasAlert’s utility in evaluating text completion and question-answering across nine different LLMs.

Key Results and Insights

Superior Detection Performance

On the RedditBias dataset, BiasAlert achieved an Accuracy of 0.84 and an F1-score of 0.82. This performance significantly exceeds that of GPT-4-as-a-Judge (Acc: 0.61, F1: 0.59) and specialized moderation tools like LlamaGuard (Acc: 0.59). Notably, BiasAlert achieved a 1.0 Classification Score in specific tests, demonstrating perfect accuracy in identifying the type of bias (e.g., racial vs. gender) once a bias was detected.

The Necessity of External Knowledge

A key finding of this research is that internal knowledge is insufficient for reliable bias detection. Even state-of-the-art models like GPT-4 struggle to identify bias accurately when relying solely on internal parameters. Ablation studies confirmed that the retrieval component (RE) provided the most significant performance boost, shifting the task from subjective "guessing" to objective comparison against human-annotated ground truths.

Real-time Mitigation and Efficiency

BiasAlert is not only an evaluation tool but also an effective guardrail. In application studies, using BiasAlert as a filter for Alpaca-7b reduced the model's "Bias Level" from 0.283 to 0.042. The tool is computationally efficient, processing a single generation in approximately 1.4 seconds on standard hardware (2x RTX 3090s), making it viable for production environments.

Limitations and Future Work

While BiasAlert represents a significant step forward, several challenges remain. Social norms evolve, meaning the retrieval database requires continuous updates to prevent "aging." Additionally, while the tool excels at catching explicit stereotypes, subtle "dog-whistle" or implicit biases remain difficult to retrieve and identify. Future work will focus on expanding the tool to handle intersectional biases (e.g., combined race and gender bias) and integrating more diverse, large-scale datasets to improve retrieval quality.

Summary updated

Abstract

Evaluating the bias in Large Language Models (LLMs) becomes increasingly crucial with their rapid development. However, existing evaluation methods rely on fixed-form outputs and cannot adapt to the flexible open-text generation scenarios of LLMs (e.g., sentence completion and question answering). To address this, we introduce BiasAlert, a plug-and-play tool designed to detect social bias in open-text generations of LLMs. BiasAlert integrates external human knowledge with inherent reasoning capabilities to detect bias reliably. Extensive experiments demonstrate that BiasAlert significantly outperforms existing state-of-the-art methods like GPT4-as-A-Judge in detecting bias. Furthermore, through application studies, we demonstrate the utility of BiasAlert in reliable LLM bias evaluation and bias mitigation across various scenarios. Model and code will be publicly released.
BibTeX
@inproceedings{fan2024biasalert,
  author = {Zhiting Fan and Ruizhe Chen and Ruiling Xu and Zuozhu Liu},
  title = {{BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.820},
}
Figure