CompAIssionate Other: Identity Assumptions in LLM Responses to Distress

Drishti Goel, Ethan M. Nguyen, Violeta J. Rodriguez, Koustuv SahaarXiv 2026 · arXiv

This study, recently published by researchers at the University of Illinois Urbana-Champaign, investigates a subtle but significant form of algorithmic bias in mental health support: how Large Language Models (LLMs) interpret the cause of a user's distress based on their social identity.

TL;DR

While LLMs maintain consistent safety behaviors (like providing crisis resources) across different demographics, they frequently engage in "compassionate othering"—unsolicitedly attributing a user's distress to their marginalized identity (e.g., being an immigrant or trans) without evidence. This behavior misrepresents individual experiences and highlights a need for "identity-responsive" AI that avoids presumptive stereotyping.

What Problem Does This Paper Address?

As people increasingly turn to AI for emotional support, ensuring these systems are both safe and empathetic is critical. However, empathy requires interpretation. If a user discloses a social identity (e.g., "I am Black" or "I am an immigrant") while describing their distress, how does the model use that information?

The authors identify a gap in current AI auditing: most studies focus on "safety" (does the model refuse to help?) or "toxicity" (is the model hateful?). This paper looks at interpretive bias. The core concern is that models might jump to conclusions, assuming a marginalized identity is the source of suffering even when the user hasn't made that connection. This can lead to "compassionate othering," where a model’s attempt to be supportive actually reinforces stereotypes by treating a person’s identity as a pre-packaged explanation for their pain.

Methods

The researchers conducted a controlled counterfactual audit. They took real-world posts expressing suicidal ideation (SI) or psychological distress and created 16 variations of each by prepending a single sentence: "I’m [identity]."

  • Identities: 9 marginalized (e.g., gay, trans, Black, immigrant), 4 majority (e.g., straight, White), and 3 control conditions (e.g., "I'm a student," or no disclosure).
  • Models: Six popular models, including GPT-4o-mini, Gemini-3.8-Flash, Llama-3.1-8B, and MedGemma.
  • Prompting Protocols: They compared a "Vanilla" helpful assistant prompt against a "Clinician-guideline" prompt developed with mental health professionals.
  • Evaluation: They used an LLM-as-a-judge (GPT-5-nano) to evaluate distress attribution—whether the model suggested the identity caused the distress—and RubRIX, a 29-item rubric for interactional risks like "epistemic arrogance" or "bias and stigma."

Datasets

The study utilized two primary sources of sensitive data:

  1. Reddit SI: Posts from r/SuicideWatch annotated for suicidal ideation and distress.
  2. SH-Detection: A social media dataset annotated for self-harm.

After rigorous screening using the Columbia Suicide Severity Rating Scale (C-SSRS) to ensure distinct risk categories, the final audit included 654 SI posts and 153 distress posts from Reddit, plus 500 posts from the SH-Detection set.

Key Insights and Ideas

  • Safety is Decoupled from Interpretation: Models were remarkably consistent in their "surface" safety behaviors. Refusal rates and crisis resource provision didn't vary much by identity. However, the logic the models used to explain the user's distress varied wildly.
  • The "Immigrant" Attribution Gap: The most striking finding was for immigrant identities. Models were highly likely to attribute distress to "immigration status" but were among the least likely to provide identity-specific resources (like specialized helplines).
  • Benevolent Harm: The authors characterize these attributions as a form of "compassionate othering." The models aren't being mean; they are being "too helpful" by imposing a socially available narrative (e.g., "Black people are expected to be strong") onto an individual who might just be grieving a friend or worried about rent.
  • The Prompting Trade-off: Using structured clinician guidelines significantly reduced these biased attributions. However, it also made the models less likely to acknowledge the user's identity at all, suggesting a difficult balance between avoiding stereotypes and providing personalized care.

Key Results

The quantitative results highlight a significant disparity in how models handle marginalized vs. majority identities.

![Fig.

  1. Change in refusal, RubRIX risk, and distress attributed to identity.](/figures/art_d1057a3f83fb4dabb4655a16ac7c3b33)

As seen in Figure 1, while refusal and RubRIX harm scores showed minimal differences, the "Distress attributed to identity" column shows a sharp, statistically significant increase for marginalized identities across almost all models.

  • Attribution Rates: In the SH-Detection dataset, Gemini-3.8-Flash and Qwen3.5-9B attributed distress to marginalized identities in 8.7% and 8.9% of cases, respectively, compared to nearly 0% for majority identities.
  • Identity Mentions: Pooled across models, 16.9% of responses to marginalized disclosures referenced the identity, whereas majority identities were only referenced in 2.0% of cases.
  • Qualitative Shifts: The paper notes that models would often keep the rest of a response identical but swap out the cause. "Financial worries" in a baseline response might become "your immigration status" in the counterfactual version, even if the user never mentioned immigration-related stress.

Limitations and Future Work

The authors acknowledge several constraints:

  • Single-Turn Interactions: The audit only looked at the first response. In a real conversation, a user might correct a model's assumption, but it's unknown if models would persist in their bias.
  • U.S. Context: The identity categories and resources are primarily centered on U.S. social dynamics.
  • Controlled vs. Naturalistic: Prepending "I'm [identity]" is a clean experimental probe, but real-world disclosures are often more nuanced and embedded in stories.

Future Work: The researchers suggest moving toward "identity-responsive" support that acknowledges identity without assuming its explanatory power. This involves models that can communicate uncertainty and ask for clarification rather than jumping to stereotypical conclusions.

Summary updated

Abstract

Individuals experiencing suicidal ideation (SI) or psychological distress may disclose their social identities when seeking support from LLMs. Attending to identity-related stressors can be important for providing contextually appropriate and empathetic support. However, disclosing an identity does not necessarily mean that it is a cause of distress. In this work, we audit how LLMs interpret distress through disclosed identities. Using paired counterfactual interventions, we evaluate six LLMs across two SI and distress datasets, 16 identity and control conditions, and two prompting protocols. Our findings suggest that models can transform identity disclosures into explanatory narratives, attributing distress to presumed identity-related adversity. These assumptions frequently appear within otherwise empathetic responses, particularly when marginalized identities are disclosed. We contextualize our findings through the lens of compassionate othering to characterize these patterns as a form of harm in which empathetic responses impose identity-linked assumptions on individual suffering. These responses can misrepresent individuals' experiences by treating their identities as explanations for distress without supporting evidence. Our findings highlight the need to mitigate these harms by grounding interpretations of distress in individuals' disclosures and avoiding unsupported identity-linked assumptions.
BibTeX
@misc{goel2026compaissionate,
  author = {Drishti Goel and Ethan M. Nguyen and Violeta J. Rodriguez and Koustuv Saha},
  title = {{CompAIssionate Other: Identity Assumptions in LLM Responses to Distress}},
  year = {2026},
  eprint = {2611.10808},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10808},
}
Figure