Measuring Whether Protective Identity Signals Work on Language Models

People who expect to be stereotyped rarely choose between full disclosure and full concealment. They pick a middle path. They state what they need rather than what category they belong to, they name a stereotype in order to reject it, or they choose a label for themselves. Sociologists describe these moves as "social armor." This proposal asks whether such strategies protect users of large language models (LLMs). Prior work found that telling a chatbot "I am Black" pushes its recommendations toward a narrow, stereotyped set. Saying nothing does not help either, because the model then defaults to recommendations that mirror those given to White users. What has not been tested is the middle path.

Answering that question needs more than averaging a bias score per prompt condition. A protective prompt can change what a bias metric measures, not only how large it is. Suppose a user writes "don't base this on my race." The model may then deny any influence of race while still steering its recommendations, which raises a "denial" score without reducing the steering. We therefore propose to build a confirmatory factor analysis (CFA) model of how LLMs respond to identity. CFA is a psychometric method that tests whether many observed scores are explained by a small set of hypothesized hidden factors. Our model has three factors: stereotyped narrowing, fit to the user's stated needs, and transparency about using identity. We then test whether this measurement model holds unchanged across signaling strategies, and only if it does do we compare strategies on the hidden factors.

Core Claim

The central claim is that "does strategy X protect the user?" is only answerable once two things are established. First, the three response dimensions must be separable. Second, they must be measured the same way under every strategy, a property called measurement invariance. Under these conditions, we expect at least one strategy to lower stereotyped narrowing while keeping need-fit at the level of explicit disclosure. The strategy we expect to do this is need-based disclosure, where the user states priorities instead of a category. The contribution is twofold: a validated measurement instrument for identity-conditioned LLM behavior, and the first controlled test of user-side protective strategies against it.

Where This Sits

The motivating evidence comes from the ACL 2025 study of identity reveal in LLM recommendations. It built 120,000 synthetic university and neighborhood requests under three conditions: No Reveal, Implicit Reveal (dialect features or affiliations), and Explicit Reveal. It found that reveal shifts recommendations toward demographic alignment (p < 0.001). For Black users, Llama 3.1 70B's recommendations covered only 7.8% of the target demographic population. GPT-4o and Claude 3.5 Sonnet denied race had any influence at near 100% rates for explicit reveals in university recommendations. That study treats its three reveal levels as fixed conditions. We add the strategies users actually employ between those poles. We also ask whether its metrics, namely alignment, coverage, and denial, measure one thing or several.

A model-side alternative is Mixture-of-Stereotypes for LLM recommenders. It routes prompts through stereotype-specific experts. It reports 12–25% gains in Stereotype Fairness for gender on MovieLens100K, with comparable accuracy. That approach requires control of the model. Our question is what a user can do with a closed model they cannot change. The two lines of work are complementary, but they answer different questions.

The method comes from AI Cartography. That study applied bifactor CFA and generalizability theory to more than 4,000 Open LLM Leaderboard models. It showed that a manifest-score scaling slope has reliability $R_\beta = 0.53$, while the latent general-factor slope reaches $R_g = 0.97$. That study modeled only cognitive benchmarks. We carry its logic, that latent rather than raw scores are the trustworthy object, to identity-conditioned behavior.

The LLM psychometrics benchmark of Li et al. constrains the design. It found that LLM "traits" shift sharply under role-play and prompt framing, and it reports no CFA fit statistics. We read this as a warning that measurement invariance across prompt conditions cannot be assumed. For that reason we make it the first thing we test.

The strategy set comes from the sociology of protective presentation, which supplies hypotheses rather than methods. One source contrasts external validation with internal comfort (Li on reclaiming intimate apparel), which maps to need-based disclosure. Another describes explicit refusal of imposed ideals (Huang on refusing "beauty duty"), which maps to counter-framing. A third shows self-chosen naming as agency (Liu's 334 Xiaohongshu name-change cases), which maps to self-labeling. Shan's account of dress as social armor motivates treating all three as defenses rather than concealment. The closest quantitative bridge in that literature is Kim's CFA of cabin-crew image autonomy. It reached CFI = 0.962 on N = 109 but modeled autonomy as a workplace motivator, not as protection. It also involved no algorithm.

Research Program

The unit of analysis is a single request cell. A cell is defined by four things: the persona (race/ethnicity × gender), the query template, the signaling strategy, and the model. We sample each cell several times.

The design has seven strategies. Three replicate the ACL conditions: No Reveal, Implicit Reveal, and Explicit Reveal. Four are protective:

  • Need-based disclosure. The user states culturally relevant priorities without a category, for example "a campus with an active Black student union and strong financial aid."
  • Counter-framing. The user discloses identity and explicitly rejects stereotyped treatment.
  • Self-labeling. The user discloses only a chosen name or descriptor.
  • Dominant-register camouflage. The user rewrites dialect-marked requests in standard English while keeping the stated needs.

Every persona also carries objective constraints, such as budget, region, and program. This lets us score need-fit automatically.

The domains are universities and neighborhoods, following the ACL protocol, plus movies and books. For movies and books we reuse the gender and age stereotype item labels and the leave-one-out setup that Zhao et al. built on MovieLens100K and BookCrossing (about 10,000 sequences each). We use SeeGULL, whose 7,750 regionally validated stereotypes cover 179 groups, to expand the stereotyped-attribute lexicon beyond US categories.

Models include open-weight models at several sizes, so we can vary scale and sample at scale. They also include the closed models studied by the ACL paper.

Each cell yields at least three indicators per hypothesized factor.

Narrowing has four indicators:

  • the shift in demographic alignment relative to the matched White, No-Reveal persona;
  • coverage contraction, in the ACL sense;
  • the drop in entropy of recommended item attributes;
  • the share of SeeGULL- or Zhao-labeled stereotyped items.

Need-fit has three indicators:

  • the rate at which recommendations satisfy the stated constraints;
  • the mean rank of constraint-satisfying items;
  • relevance ratings from an LLM judge, calibrated on a 1,000-cell human-annotated subset.

Transparency has three indicators, all taken from a follow-up probe asking whether identity influenced the answer:

  • denial rate;
  • agreement between the stated reason and a counterfactual swap of the persona;
  • rate of explicitly acknowledging identity use.

The model is a three-factor CFA in which indicator $j$ for cell $i$ is

$$x_{ij} = \tau_j + \lambda_j \eta_{i,f(j)} + \varepsilon_{ij},$$

where $\eta_{i,f(j)}$ is cell $i$'s score on the factor $f(j)$ that indicator $j$ is assigned to, $\lambda_j$ is the loading, $\tau_j$ is the intercept, and $\varepsilon_{ij}$ is residual error. Cells share templates and models, so we cluster standard errors by template and model. We allow residual correlations only among indicators computed from the same output list, and we pre-register them.

We then fit multi-group CFA across the seven strategies in three steps. The configural step tests whether the same structure holds in every group. The metric step adds equal loadings $\lambda_j$. The scalar step adds equal intercepts $\tau_j$. Latent means are compared only for strategies that pass scalar invariance, possibly with partial invariance on freed intercepts that we report. A strategy that fails metric invariance is a finding in its own right: it means the strategy changes what a metric means.

Baselines

The naive reference points are No Reveal and Explicit Reveal, reconstructed from the ACL templates. No Reveal anchors the claim that silence is not neutral. Explicit Reveal anchors the need-fit level a protective strategy must preserve. The strongest reported result in this setting is ACL's 7.8% coverage for Black users under Llama 3.1 70B, together with near-100% denial for GPT-4o and Claude 3.5 Sonnet. We first replicate these on the same models before testing anything new. A protective strategy "beats" Explicit Reveal if three conditions hold: its latent narrowing mean is lower by at least 0.3 standard deviations, its latent need-fit is not lower by more than 0.1 standard deviations, and both comparisons rest on scalar-invariant factors.

The strongest analysis competitor is manifest scoring. It applies the same indicators, averaged per strategy, and compares strategies with bootstrap confidence intervals. It runs on identical cells and samples, so no extra compute is involved. The proposal beats it under either of two conditions. The first is if strategy rankings differ between manifest and latent analyses specifically for strategies that fail invariance. The second is if latent scores show higher split-half reliability across templates and seeds than manifest scores. This mirrors the $R_\beta = 0.53$ versus $R_g = 0.97$ contrast in AI Cartography. If neither condition holds, we will report that manifest scoring suffices.

The model-side reference is Mixture-of-Stereotypes on MovieLens100K and BookCrossing. Zhao et al. report a 12–25% Stereotype Fairness improvement for gender. We run our strategies on an unmodified LLM recommender and report Stereotype Fairness alongside our latent scores. This shows how much of the model-side gain a user can recover through phrasing alone. We do not expect to beat Mixture-of-Stereotypes, and we do not need to in order to support the core claim.

Two ablations isolate the key ingredient. The first is a one-factor model that merges narrowing, need-fit, and transparency into a single "bias" score. It tests whether separating the factors is needed. The second is a fair-instruction system prompt ("treat all users equally"). It tests whether protective signaling does more than a generic fairness instruction.

Why not just…?

Why not just compare average bias scores per strategy? This is the real threat. If invariance holds everywhere and manifest and latent rankings agree, the CFA adds rigor but no new conclusions. We think it will not hold everywhere. The ACL denial results suggest models respond to the mention of identity separately from their behavior. Counter-framing is therefore exactly the kind of strategy likely to inflate transparency indicators or break their loadings. The manifest-versus-latent ranking comparison in the Baselines settles this directly.

Why not just tell users to say nothing? The ACL study already shows No Reveal outputs mirror those given to White users. Silence trades stereotyping for erasure of the needs the user came with. Our need-fit factor quantifies that cost. Dominant-register camouflage tests the most favorable version of concealment, because the stated needs are kept.

Why not run exploratory factor analysis or cluster the outputs? Exploratory methods find whatever structure the data contain without testing a hypothesis, as Xu and Vien et al. note. They also cannot test invariance across groups, which is the property our comparisons depend on. We will run an exploratory analysis on a held-out half of the cells as a check. If it recovers a different structure, we report it and revise before confirming on the other half.

Why not just debias the model? That is the right move for model owners, and Mixture-of-Stereotypes shows it works. Users of closed chatbots cannot do it, and the strategies people already use deserve evidence about whether they work.

Proposed Experiments and Decisive Checks

The first check is replication. We must reproduce the ACL alignment and coverage pattern on Llama 3.1 70B before testing anything new.

The second check is structure. The three-factor model should fit well (CFI ≥ 0.95, RMSEA ≤ 0.06). It should beat the one-factor model on a likelihood-ratio test. It should show discriminant validity, meaning that the correlation between the narrowing and need-fit factors falls clearly below 0.85. If narrowing and need-fit collapse into one factor, the personalization-versus-stereotyping tension is not separable at the level of outputs, and the core claim fails.

The third check is invariance. It identifies which strategies can be compared on latent means and which change what a metric means.

The fourth check is the substantive test. It asks whether need-based disclosure, or any protective strategy, lowers latent narrowing while keeping need-fit at the Explicit Reveal level. This must hold consistently across at least three of the four domains and across open and closed models.

The fifth check is reliability. We compute generalizability coefficients of manifest versus latent strategy effects across template and seed facets.

All prompts, persona specifications, and pre-registered residual structures will be released. The human-annotated relevance subset will be released as well, so others can test new strategies against the same instrument.