Identity or Prompt Noise? A Calibrated Invariance Audit of LLM Code Generation

Maksim E. Eren, Ryan Barron, Eric Michalak, Charles Nicholas, Manish BhattaraiarXiv 2026 · arXiv

TL;DR

By auditing over 30 million LLM-generated Python programs, this paper reveals that task-irrelevant personas (like gender, country, or occupation) cause small, reproducible changes in code form (style, length, comments) but do not reliably impact functional correctness. Crucially, the authors demonstrate that without rigorous statistical calibration, random prompt noise and sample size imbalances can easily be mistaken for systemic demographic bias.

What Problem Does This Paper Address?

When using Large Language Models (LLMs) for code generation, the output should ideally depend entirely on the programming specification. If you add task-irrelevant context to a prompt—such as assigning the model a persona based on gender, country, or occupation—the functional correctness and structural implementation of the code should remain invariant.

While prior work has explored demographic stereotypes and discriminatory logic in LLMs, this paper tackles irrelevant-identity invariance. The challenge is that LLMs are notoriously sensitive to prompt wording. If a model generates slightly different code when prompted as a "doctor" versus a "carpenter," is it due to a deeply encoded social bias, or is it just random statistical noise from innocuous rewording? The authors aim to separate true identity-conditioned variation from baseline prompt noise.

Methods

The authors designed a large-scale counterfactual invariance audit. They held programming tasks fixed while systematically varying assigned persona attributes across three axes: gender, country, and occupation.

To distinguish meaningful variation from random noise, they introduced a within-task randomization test:

  1. Observed Dispersion: For each model-benchmark-axis combination, they calculated the standard deviation of identity-level means (centered within each task).
  2. Permutation Baseline: They shuffled the identity labels within each task 1,000 times. This broke the connection between the specific identity and the output while preserving the exact sample sizes and underlying outcome distributions.
  3. Calibration: They computed an observed-to-null ratio and an "excess dispersion" metric. If the observed variation didn't exceed this shuffled baseline, the variation was deemed a byproduct of sample allocation and prompt noise rather than a structural identity effect.

Primary outcomes were evaluated using EvalPlus pass rates (for functional correctness) and CodeBLEU (for structural form, combining lexical, syntactic, and data-flow similarity). They also applied Benjamini–Hochberg False Discovery Rate (FDR) corrections to control for multiple testing.

Datasets

The evaluation leveraged two popular augmented functional testing benchmarks:

  • HumanEval+ (164 tasks)
  • MBPP+ (378 tasks)

These tasks were paired with an extensive set of identity labels:

  • 20 gender labels
  • 112 countries
  • 234 occupations
  • 10 generic persona-noun controls (to isolate wording effects)

The main archive consisted of 30.73 million executed Python generations across seven model checkpoints: Llama-3.2-3B, Mistral-7B, Llama-3-8B, Nemotron-3-Nano-30B, Gemma-4-31B, gpt-oss-120B, and Nemotron-3-Super-120B.

Key Insights and Ideas

  • Calibration Reverses Raw Conclusions: If you look only at uncalibrated, raw data, country appears to cause the most variance in code generation. However, this is an illusion caused by smaller sample sizes per country. After calibration against the randomized null baseline, country effects virtually disappeared, while occupation emerged as the most consistent structural driver of code changes.
  • Form Changes, Function Doesn't: Assigned personas subtly alter how code is written (e.g., choice of tokens, number of lines, density of comments), but they rarely change whether the code works.
  • The Wording Confound: Gender-associated variations could not be cleanly separated from the specific wording of the persona nouns themselves. When regrouping gender rows purely by the underlying noun phrasing, the variations matched the "gender" effect, suggesting the model reacts to surface-level text patterns rather than abstract demographic concepts.
  • No Systemic Disadvantage: The data does not support stable favoritism or harm toward specific named identities. An occupation that saw lower pass rates in one model might perform better in another; rankings were highly unstable across different checkpoints.

Key Results

  • Occupation Dominates Structure: Occupation CodeBLEU dispersion survived FDR correction in 10 out of 14 main model-benchmark cells. However, the actual effect size was remarkably small, with a median excess dispersion of just 0.141 points.
  • Country Effects Evaporate: While country led raw, uncalibrated dispersion in 12 of 14 cells, its median calibrated ratio dropped to 1.00 (indicating no effect beyond random noise).
Calibrated identity sensitivity across different models and benchmarks. Asterisks indicate statistically significant cells after FDR correction.
  • Consistency Across Structural Metrics: In high-pass-rate models, occupation-conditioned changes consistently replicated across multiple independent structural metrics (token BLEU, lines of code, comment lines, and complexity), even while functional pass rates remained statistically unaffected.
Table 2. Occupation sensitivity in six full, high-pass-rate cells.

Limitations and Future Work

The authors note several boundaries to their findings:

  1. Token-Level vs. Semantic Effects: The permutation test evaluates label exchangeability within a task, not the stochastic variation that comes from repeatedly decoding the exact same prompt.
  2. Incomplete Crossing: Persona nouns and occupations were only partially crossed in the prompt design, limiting the ability to completely isolate semantic identity from wording.
  3. Scope of Harm: This study focused on short, single-turn code generation benchmarks. It did not evaluate potential biases in longer conversational contexts, code security vulnerabilities, actual developer maintainability, or educational settings (e.g., whether an LLM provides worse explanations to certain user personas over time).

Future work should focus on testing fully crossed paraphrases, repeated exact prompts, and downstream impacts on software security and maintenance.

Summary updated

Abstract

Identity cues are irrelevant to a fixed programming specification, but raw counterfactual differences can arise from unequal samples and prompt wording. We audit 30.73 million executed Python generations from seven checkpoints on HumanEval+ and MBPP+, supplemented by an exploratory 550B slice, under model-assigned gender, country, and occupation personas. Within-task randomization and false-discovery-rate control identify occupation as the most consistent structural axis: CodeBLEU dispersion exceeds its exchangeability null in 10/14 model--benchmark cells, remains significant in 8/12 full-coverage cells, and exceeds the country ratio in every paired cell, although the median excess is only 0.141 points. In six high-pass-rate cells, occupation dispersion replicates across token similarity, length, comments, reference similarity, and complexity, while pass-rate dispersion is significant in none. Country leads raw dispersion in 12/14 cells but has a median calibrated ratio of 1.00. Gender-associated variation cannot be separated from persona wording in this design. Thus, the evidence supports small, reproducible occupation-conditioned changes in code form, not stable disadvantage to named identities or demonstrated downstream harm. We use this finding to motivate a broader commentary on the potential impacts of bias in code generation, including the possibility that models may condition their outputs on identity information available from prior conversational context.
BibTeX
@misc{eren2026identity,
  author = {Maksim E. Eren and Ryan Barron and Eric Michalak and Charles Nicholas and Manish Bhattarai},
  title = {{Identity or Prompt Noise? A Calibrated Invariance Audit of LLM Code Generation}},
  year = {2026},
  eprint = {2609.22511},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.22511},
}
Figure