TL;DR
By auditing over 30 million LLM-generated Python programs, this paper reveals that task-irrelevant personas (like gender, country, or occupation) cause small, reproducible changes in code form (style, length, comments) but do not reliably impact functional correctness. Crucially, the authors demonstrate that without rigorous statistical calibration, random prompt noise and sample size imbalances can easily be mistaken for systemic demographic bias.
What Problem Does This Paper Address?
When using Large Language Models (LLMs) for code generation, the output should ideally depend entirely on the programming specification. If you add task-irrelevant context to a prompt—such as assigning the model a persona based on gender, country, or occupation—the functional correctness and structural implementation of the code should remain invariant.
While prior work has explored demographic stereotypes and discriminatory logic in LLMs, this paper tackles irrelevant-identity invariance. The challenge is that LLMs are notoriously sensitive to prompt wording. If a model generates slightly different code when prompted as a "doctor" versus a "carpenter," is it due to a deeply encoded social bias, or is it just random statistical noise from innocuous rewording? The authors aim to separate true identity-conditioned variation from baseline prompt noise.
Methods
The authors designed a large-scale counterfactual invariance audit. They held programming tasks fixed while systematically varying assigned persona attributes across three axes: gender, country, and occupation.
To distinguish meaningful variation from random noise, they introduced a within-task randomization test:
- Observed Dispersion: For each model-benchmark-axis combination, they calculated the standard deviation of identity-level means (centered within each task).
- Permutation Baseline: They shuffled the identity labels within each task 1,000 times. This broke the connection between the specific identity and the output while preserving the exact sample sizes and underlying outcome distributions.
- Calibration: They computed an observed-to-null ratio and an "excess dispersion" metric. If the observed variation didn't exceed this shuffled baseline, the variation was deemed a byproduct of sample allocation and prompt noise rather than a structural identity effect.
Primary outcomes were evaluated using EvalPlus pass rates (for functional correctness) and CodeBLEU (for structural form, combining lexical, syntactic, and data-flow similarity). They also applied Benjamini–Hochberg False Discovery Rate (FDR) corrections to control for multiple testing.
Datasets
The evaluation leveraged two popular augmented functional testing benchmarks:
- HumanEval+ (164 tasks)
- MBPP+ (378 tasks)
These tasks were paired with an extensive set of identity labels:
- 20 gender labels
- 112 countries
- 234 occupations
- 10 generic persona-noun controls (to isolate wording effects)
The main archive consisted of 30.73 million executed Python generations across seven model checkpoints: Llama-3.2-3B, Mistral-7B, Llama-3-8B, Nemotron-3-Nano-30B, Gemma-4-31B, gpt-oss-120B, and Nemotron-3-Super-120B.
Key Insights and Ideas
- Calibration Reverses Raw Conclusions: If you look only at uncalibrated, raw data, country appears to cause the most variance in code generation. However, this is an illusion caused by smaller sample sizes per country. After calibration against the randomized null baseline, country effects virtually disappeared, while occupation emerged as the most consistent structural driver of code changes.
- Form Changes, Function Doesn't: Assigned personas subtly alter how code is written (e.g., choice of tokens, number of lines, density of comments), but they rarely change whether the code works.
- The Wording Confound: Gender-associated variations could not be cleanly separated from the specific wording of the persona nouns themselves. When regrouping gender rows purely by the underlying noun phrasing, the variations matched the "gender" effect, suggesting the model reacts to surface-level text patterns rather than abstract demographic concepts.
- No Systemic Disadvantage: The data does not support stable favoritism or harm toward specific named identities. An occupation that saw lower pass rates in one model might perform better in another; rankings were highly unstable across different checkpoints.
Key Results
- Occupation Dominates Structure: Occupation CodeBLEU dispersion survived FDR correction in 10 out of 14 main model-benchmark cells. However, the actual effect size was remarkably small, with a median excess dispersion of just 0.141 points.
- Country Effects Evaporate: While country led raw, uncalibrated dispersion in 12 of 14 cells, its median calibrated ratio dropped to 1.00 (indicating no effect beyond random noise).
- Consistency Across Structural Metrics: In high-pass-rate models, occupation-conditioned changes consistently replicated across multiple independent structural metrics (token BLEU, lines of code, comment lines, and complexity), even while functional pass rates remained statistically unaffected.
Limitations and Future Work
The authors note several boundaries to their findings:
- Token-Level vs. Semantic Effects: The permutation test evaluates label exchangeability within a task, not the stochastic variation that comes from repeatedly decoding the exact same prompt.
- Incomplete Crossing: Persona nouns and occupations were only partially crossed in the prompt design, limiting the ability to completely isolate semantic identity from wording.
- Scope of Harm: This study focused on short, single-turn code generation benchmarks. It did not evaluate potential biases in longer conversational contexts, code security vulnerabilities, actual developer maintainability, or educational settings (e.g., whether an LLM provides worse explanations to certain user personas over time).
Future work should focus on testing fully crossed paraphrases, repeated exact prompts, and downstream impacts on software security and maintenance.
Summary updated