Same Facts, Different Remedy: Linguistic Profiling and Arbitrariness in LLM and System One Decisions

Colin PriestarXiv 2026 · arXiv

TL;DR

This paper audits Large Language Models (LLMs) used for administrative decision-making, revealing that while systematic group-level bias (linguistic profiling) is relatively small, individual-level inconsistency (arbitrariness) is a much larger problem. The study demonstrates that "how" a complaint is written can change the outcome 2.6–6.3% of the time, even when the underlying facts remain identical.

What Problem Does This Paper Address?

As LLMs are increasingly deployed to triage and resolve consumer complaints, a critical fairness question arises: does the model respond to the content of the complaint or the style in which it is written?

Sociolinguistic research shows that writing styles vary systematically with age, education, and socioeconomic status (SES). If a model gives more favorable remedies to a college-educated writer than a high-school-educated one for the same grievance, it is practicing "linguistic profiling."

The author identifies a major gap in current fairness audits: they often conflate profiling (systematic bias against groups) with arbitrariness (random instability where a model changes its mind based on minor wording tweaks). This paper seeks to disentangle these two phenomena and evaluate whether common mitigations, like "style standardization," actually work.

Methods

The researcher employed a matched-guise technique, a classic sociolinguistic method adapted for AI.

  1. Rewriting: 1,000 complaints from a public database were rewritten into 36 different "guises" (combinations of age, geography, education, and SES) by two independent LLMs, with strict instructions to preserve every fact.
  2. Auditing: Three systems were tested: two generative LLMs (GPT-4o-mini and GPT-5-mini) and one probabilistic "System One" model (Jev).
  3. Controls: To isolate the cause of variation, the study controlled for option order, inference noise (repeated scoring), and paraphrase effects.
  4. Mitigation: The study tested "style standardization"—rewriting all complaints into a neutral, professional style before the final decision-making step.
  5. Fact-Fidelity Audit: External LLMs (Claude and a different GPT version) acted as "blind" auditors to ensure the rewrites didn't accidentally change the material facts of the complaints.

Datasets

  • CFPB Consumer Complaint Database: A public dataset of US consumer financial complaints.
  • Synthetic Matched-Guise Panels: 72,000 unique rewrites (for the holdout set) generated specifically for this study to simulate different demographic writing styles.

Key Insights and Ideas

  • The Two Scales of Fairness: Bias exists at two levels. At the group scale, there is a small but reliable "generosity" bias toward older, college-educated, and higher-SES writing styles. However, at the individual scale, the model's "arbitrariness" is far more impactful, leading to different decisions for the same person depending on their word choice.
  • Delivery vs. Content: Using Gelbach’s decomposition, the author found that bias is driven 4–7× more by "delivery" (vocabulary, grammar, politeness) than by "content" (the actual facts of the case).
  • The "Wording Lottery": Even when a model is 387× more sensitive to facts than to style (a "positive control" showing the model isn't just guessing), the incidental choice of words still creates significant instability.
  • Standardization is a Double-Edged Sword: Rewriting complaints into a "standard" style successfully reduced group-level gaps by up to 94%. However, it did not make individual decisions more stable and, in some cases, actually reversed the direction of the bias.

Key Results

The study highlights a stark contrast between group-level disparities and individual-level instability.

Figure 1: Two scales (held-out panel, n = 1,000). (a) Group persona gaps in the monetary- remedy rate, with the ±1 pp benchmark. (b) Individual instability: the probability that two style versions of the same complaint receive different monetary decisions.
  • Group Gaps: Every monetary-remedy gap was within 0.5 percentage points—small enough to pass common regulatory "four-fifths" rules.
  • Individual Instability: Two versions of the same complaint received different monetary decisions 2.6% to 6.3% of the time. GPT-5-mini was the most "arbitrary," with a 6.3% disagreement rate.
  • Mitigation Performance: Style standardization reduced GPT-4o-mini's SES gap from +0.80 pp to +0.42 pp, but it failed to significantly lower the probability of disagreeing remedies for individuals.
Table 1: Monetary decisions under explicit remedy rules (held-out panel, post hoc). Rate: monetary-decision rate (%). Differ: probability that two style versions of the same complaint receive different monetary decisions (%, 95% CI). Fact span: strengthen −weaken monetary rate (pp, GPT-4o-mini bank, 200 complaints).

Limitations and Future Work

  • Synthetic Guises: The "personas" were generated by LLMs, which might rely on stereotypes of how different groups write rather than capturing the full nuance of real human diversity.
  • Fact Preservation: Despite instructions, independent auditors found that 3.3% to 11.7% of rewrites contained material factual changes, suggesting that "fact-preserving" LLM rewrites are not yet perfect.
  • Narrow Scope: The study focused on US financial complaints in English. Future work should explore whether these findings hold across different languages, cultures, and administrative contexts (like healthcare or legal triage).
  • No "Ground Truth": The study measures sensitivity and consistency, not accuracy. We don't know which decision was "correct," only that the model changed its mind.

Summary updated

Abstract

Large language models are increasingly used to resolve consumer complaints, and the way complainants write varies with age, education and socioeconomic status (SES). We audit three remedy-decision systems using sociolinguistic rewrites of US consumer financial complaints that are instructed to preserve every fact: two generative LLMs that return a single label (GPT-4o-mini and GPT-5-mini) and a probabilistic "System One" model that returns a distribution over remedies (Jev). Using pre-registered hypotheses tested once on held-out complaints, we separate two phenomena that single-label audits conflate. At the group scale, older, college-educated and higher-SES styles receive slightly more generous remedies; these effects replicate but are small, with every monetary-remedy gap within 0.5 percentage points. At the individual scale, two style versions of the same complaint receive different monetary decisions 2.6-6.3% of the time. In a variance-component analysis of the probabilistic model, persona components shared across the two rewrite models account for about 1.6% of within-complaint variation. The persona gaps are associated mainly with delivery (vocabulary, grammar, politeness) rather than content. Positive controls show that all three systems respond 49-387 times more strongly to remedy-relevant facts than to style. Rewriting every complaint into one standard style reduced GPT-4o-mini's education and SES gaps by 94% and 87% while retaining fact sensitivity, but reversed two of the probabilistic model's gaps and did not make individual decisions more stable. Independent LLM auditors flagged material fact changes in 3.3% and 11.7% of rewrites, and instability persisted among unflagged rewrites. Fairness audits of LLM decision systems should report group disparities and individual instability as separate outcomes.
BibTeX
@misc{priest2026same,
  author = {Colin Priest},
  title = {{Same Facts, Different Remedy: Linguistic Profiling and Arbitrariness in LLM and System One Decisions}},
  year = {2026},
  eprint = {2611.08995},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08995},
}
Figure