TL;DR
This paper audits Large Language Models (LLMs) used for administrative decision-making, revealing that while systematic group-level bias (linguistic profiling) is relatively small, individual-level inconsistency (arbitrariness) is a much larger problem. The study demonstrates that "how" a complaint is written can change the outcome 2.6–6.3% of the time, even when the underlying facts remain identical.
What Problem Does This Paper Address?
As LLMs are increasingly deployed to triage and resolve consumer complaints, a critical fairness question arises: does the model respond to the content of the complaint or the style in which it is written?
Sociolinguistic research shows that writing styles vary systematically with age, education, and socioeconomic status (SES). If a model gives more favorable remedies to a college-educated writer than a high-school-educated one for the same grievance, it is practicing "linguistic profiling."
The author identifies a major gap in current fairness audits: they often conflate profiling (systematic bias against groups) with arbitrariness (random instability where a model changes its mind based on minor wording tweaks). This paper seeks to disentangle these two phenomena and evaluate whether common mitigations, like "style standardization," actually work.
Methods
The researcher employed a matched-guise technique, a classic sociolinguistic method adapted for AI.
- Rewriting: 1,000 complaints from a public database were rewritten into 36 different "guises" (combinations of age, geography, education, and SES) by two independent LLMs, with strict instructions to preserve every fact.
- Auditing: Three systems were tested: two generative LLMs (GPT-4o-mini and GPT-5-mini) and one probabilistic "System One" model (Jev).
- Controls: To isolate the cause of variation, the study controlled for option order, inference noise (repeated scoring), and paraphrase effects.
- Mitigation: The study tested "style standardization"—rewriting all complaints into a neutral, professional style before the final decision-making step.
- Fact-Fidelity Audit: External LLMs (Claude and a different GPT version) acted as "blind" auditors to ensure the rewrites didn't accidentally change the material facts of the complaints.
Datasets
- CFPB Consumer Complaint Database: A public dataset of US consumer financial complaints.
- Synthetic Matched-Guise Panels: 72,000 unique rewrites (for the holdout set) generated specifically for this study to simulate different demographic writing styles.
Key Insights and Ideas
- The Two Scales of Fairness: Bias exists at two levels. At the group scale, there is a small but reliable "generosity" bias toward older, college-educated, and higher-SES writing styles. However, at the individual scale, the model's "arbitrariness" is far more impactful, leading to different decisions for the same person depending on their word choice.
- Delivery vs. Content: Using Gelbach’s decomposition, the author found that bias is driven 4–7× more by "delivery" (vocabulary, grammar, politeness) than by "content" (the actual facts of the case).
- The "Wording Lottery": Even when a model is 387× more sensitive to facts than to style (a "positive control" showing the model isn't just guessing), the incidental choice of words still creates significant instability.
- Standardization is a Double-Edged Sword: Rewriting complaints into a "standard" style successfully reduced group-level gaps by up to 94%. However, it did not make individual decisions more stable and, in some cases, actually reversed the direction of the bias.
Key Results
The study highlights a stark contrast between group-level disparities and individual-level instability.
- Group Gaps: Every monetary-remedy gap was within 0.5 percentage points—small enough to pass common regulatory "four-fifths" rules.
- Individual Instability: Two versions of the same complaint received different monetary decisions 2.6% to 6.3% of the time. GPT-5-mini was the most "arbitrary," with a 6.3% disagreement rate.
- Mitigation Performance: Style standardization reduced GPT-4o-mini's SES gap from +0.80 pp to +0.42 pp, but it failed to significantly lower the probability of disagreeing remedies for individuals.
Limitations and Future Work
- Synthetic Guises: The "personas" were generated by LLMs, which might rely on stereotypes of how different groups write rather than capturing the full nuance of real human diversity.
- Fact Preservation: Despite instructions, independent auditors found that 3.3% to 11.7% of rewrites contained material factual changes, suggesting that "fact-preserving" LLM rewrites are not yet perfect.
- Narrow Scope: The study focused on US financial complaints in English. Future work should explore whether these findings hold across different languages, cultures, and administrative contexts (like healthcare or legal triage).
- No "Ground Truth": The study measures sensitivity and consistency, not accuracy. We don't know which decision was "correct," only that the model changed its mind.
Summary updated