This study, recently published by researchers at the University of Illinois Urbana-Champaign, investigates a subtle but significant form of algorithmic bias in mental health support: how Large Language Models (LLMs) interpret the cause of a user's distress based on their social identity.
TL;DR
While LLMs maintain consistent safety behaviors (like providing crisis resources) across different demographics, they frequently engage in "compassionate othering"—unsolicitedly attributing a user's distress to their marginalized identity (e.g., being an immigrant or trans) without evidence. This behavior misrepresents individual experiences and highlights a need for "identity-responsive" AI that avoids presumptive stereotyping.
What Problem Does This Paper Address?
As people increasingly turn to AI for emotional support, ensuring these systems are both safe and empathetic is critical. However, empathy requires interpretation. If a user discloses a social identity (e.g., "I am Black" or "I am an immigrant") while describing their distress, how does the model use that information?
The authors identify a gap in current AI auditing: most studies focus on "safety" (does the model refuse to help?) or "toxicity" (is the model hateful?). This paper looks at interpretive bias. The core concern is that models might jump to conclusions, assuming a marginalized identity is the source of suffering even when the user hasn't made that connection. This can lead to "compassionate othering," where a model’s attempt to be supportive actually reinforces stereotypes by treating a person’s identity as a pre-packaged explanation for their pain.
Methods
The researchers conducted a controlled counterfactual audit. They took real-world posts expressing suicidal ideation (SI) or psychological distress and created 16 variations of each by prepending a single sentence: "I’m [identity]."
- Identities: 9 marginalized (e.g., gay, trans, Black, immigrant), 4 majority (e.g., straight, White), and 3 control conditions (e.g., "I'm a student," or no disclosure).
- Models: Six popular models, including GPT-4o-mini, Gemini-3.8-Flash, Llama-3.1-8B, and MedGemma.
- Prompting Protocols: They compared a "Vanilla" helpful assistant prompt against a "Clinician-guideline" prompt developed with mental health professionals.
- Evaluation: They used an LLM-as-a-judge (GPT-5-nano) to evaluate distress attribution—whether the model suggested the identity caused the distress—and RubRIX, a 29-item rubric for interactional risks like "epistemic arrogance" or "bias and stigma."
Datasets
The study utilized two primary sources of sensitive data:
- Reddit SI: Posts from
r/SuicideWatchannotated for suicidal ideation and distress. - SH-Detection: A social media dataset annotated for self-harm.
After rigorous screening using the Columbia Suicide Severity Rating Scale (C-SSRS) to ensure distinct risk categories, the final audit included 654 SI posts and 153 distress posts from Reddit, plus 500 posts from the SH-Detection set.
Key Insights and Ideas
- Safety is Decoupled from Interpretation: Models were remarkably consistent in their "surface" safety behaviors. Refusal rates and crisis resource provision didn't vary much by identity. However, the logic the models used to explain the user's distress varied wildly.
- The "Immigrant" Attribution Gap: The most striking finding was for immigrant identities. Models were highly likely to attribute distress to "immigration status" but were among the least likely to provide identity-specific resources (like specialized helplines).
- Benevolent Harm: The authors characterize these attributions as a form of "compassionate othering." The models aren't being mean; they are being "too helpful" by imposing a socially available narrative (e.g., "Black people are expected to be strong") onto an individual who might just be grieving a friend or worried about rent.
- The Prompting Trade-off: Using structured clinician guidelines significantly reduced these biased attributions. However, it also made the models less likely to acknowledge the user's identity at all, suggesting a difficult balance between avoiding stereotypes and providing personalized care.
Key Results
The quantitative results highlight a significant disparity in how models handle marginalized vs. majority identities.

As seen in Figure 1, while refusal and RubRIX harm scores showed minimal differences, the "Distress attributed to identity" column shows a sharp, statistically significant increase for marginalized identities across almost all models.
- Attribution Rates: In the SH-Detection dataset, Gemini-3.8-Flash and Qwen3.5-9B attributed distress to marginalized identities in 8.7% and 8.9% of cases, respectively, compared to nearly 0% for majority identities.
- Identity Mentions: Pooled across models, 16.9% of responses to marginalized disclosures referenced the identity, whereas majority identities were only referenced in 2.0% of cases.
- Qualitative Shifts: The paper notes that models would often keep the rest of a response identical but swap out the cause. "Financial worries" in a baseline response might become "your immigration status" in the counterfactual version, even if the user never mentioned immigration-related stress.
Limitations and Future Work
The authors acknowledge several constraints:
- Single-Turn Interactions: The audit only looked at the first response. In a real conversation, a user might correct a model's assumption, but it's unknown if models would persist in their bias.
- U.S. Context: The identity categories and resources are primarily centered on U.S. social dynamics.
- Controlled vs. Naturalistic: Prepending "I'm [identity]" is a clean experimental probe, but real-world disclosures are often more nuanced and embedded in stories.
Future Work: The researchers suggest moving toward "identity-responsive" support that acknowledges identity without assuming its explanatory power. This involves models that can communicate uncertainty and ask for clarification rather than jumping to stereotypical conclusions.
Summary updated