Abstract
Existing debiasing techniques are typically training-based or require access to the model’s internals and output distributions, so they are inaccessible to end-users looking to adapt LLM outputs for their particular needs. In this study, we examine whether structured prompting techniques can offer opportunities for fair text generation. We evaluate a comprehensive end-user-focused iterative framework of debiasing that applies System 2 thinking processes for prompts to induce logical, reflective, and critical text generation, with single, multi-step, instruction, and role-based variants. By systematically evaluating many LLMs across many datasets and different prompting strategies, we show that the more complex System 2-based Implicative Prompts significantly improve over other techniques demonstrating lower mean bias in the outputs with competitive performance on the downstream tasks. Our work offers research directions for the design and the potential of end-user-focused evaluative frameworks for LLM use.
Introduction
Large Language Models (LLMs) often reflect and amplify societal biases present in their training data. While researchers have developed various "white-box" debiasing techniques—such as fine-tuning or adjusting output logits—these methods are frequently inaccessible to end-users who interact with models via closed-source APIs. Furthermore, retraining models is prohibitively expensive and often incurs a "fairness tax," where improvements in neutrality lead to performance drops in downstream reasoning tasks.
This study addresses these challenges by introducing a "black-box" framework for debiasing LLMs using structured prompting. Inspired by Daniel Kahneman’s "System 2" (slow, reflective) thinking process, the framework encourages models to engage in logical and critical reflection to mitigate bias without requiring access to internal model weights.
Methodology
The authors categorize prompting strategies based on their complexity and reflectiveness, moving from simple "System 1" heuristics to more complex "System 2" reasoning:
- Prefix Prompting (PP): A simple approach involving prepending a debiasing instruction (e.g., "The following text is unbiased...") or a role-based persona (e.g., "You are an unbiased person...") to the user's prompt.
- Self-Refinement (SR): A multi-step process where the model generates an initial output, reviews it, and is then asked to "self-correct" to produce a fairer version.
- Implication Prompting (IP): The most novel "System 2" approach. It follows a three-step pipeline: first, generating an initial response; second, asking the model to explicitly identify the implication or stereotype present in that response; and third, asking the model to rewrite the response to be neutral, specifically referencing the identified implication.
The framework was evaluated across four LLMs (GPT-J, Mistral, Llama-2, and MPT-Instruct) using benchmarks like StereoSet (bias), Regard (social perception), and RealToxicityPrompts, alongside downstream tasks like TruthfulQA to monitor performance degradation.
Key Results
Implication Prompting (IP) emerged as the most effective strategy across nearly all benchmarks. By forcing the model to explicitly name a stereotype, the model effectively "breaks" its statistical tendency to follow biased patterns.
- StereoSet Performance: IP achieved a 4.05% lower Stereotype Score (SS) and a 6.80% higher Idealized Context Association Test (ICAT) score compared to other methods. As shown in the results table, IP and its variants consistently lowered SS across different model architectures.
- Toxicity and Regard: IP reduced average toxicity by approximately 7% and showed a 26.85% average improvement in the Regard metric, shifting outputs toward more neutral or positive social perceptions.
- No Fairness Tax: Unlike training-based methods, these structured prompts did not degrade performance on general tasks. In fact, IP improved TruthfulQA scores by approximately 9% over base models.
Discussion and Insights
The study highlights several critical insights for prompt engineering. First, personas matter: role-based prompting (telling the model to be an unbiased person) consistently outperformed simple instructions. Second, there are diminishing returns to iteration: in Self-Refinement (SR), the most significant fairness gains occurred in the first step ($k=1$), with subsequent steps yielding negligible improvements.
While effective, Implication Prompting increases inference costs due to its multi-step nature. Furthermore, the method's success depends on the model's latent knowledge; it can only mitigate stereotypes it is capable of recognizing. Future research may explore even more complex reasoning frameworks, such as "Tree-of-Thought," to deepen these debiasing effects in larger-scale models.
Summary updated