Stereotypes as Gap-Fillers: Pragmatic Indirectness and Bias in Specialized Models

When a user says "It's chilly in here" instead of "Turn up the heat," a model has to fill in a meaning the words leave out. We propose that when a model cannot recover that meaning from context, it falls back on whatever else it knows about the speaker, and that this often means demographic stereotypes. If that is right, bias in deployed assistants is concentrated in the indirect, hint-laden requests real users actually make, not in the explicit template prompts most bias benchmarks use. The effect should be worst in compact, domain-specialized models, which are known to collapse on indirect phrasing. The proposal measures this effect and then tests a training fix. The fix teaches a specialized model to read an indirect request the way it would read the explicit paraphrase of that request, so the speaker's identity matters only when the words themselves make it matter.

Two findings motivate this. Small task-tuned dialogue models fail on indirect requests: a T5 dialogue state tracker drops from 0.51 to 0.13 accuracy when commands become indirect (Making Task-Oriented Dialogue Datasets More Natural by Synthetically Generating Indirect User Requests). Separately, models fall back on stereotypes when a task strains them: refusal of stereotyped choices erodes as task complexity rises (Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach). No one has asked whether the first failure causes the second.

Core Claim

We make three claims, and each can be checked separately. First, holding the intended meaning fixed, a model's sensitivity to an irrelevant speaker attribute (gender, age, nationality) grows as the request becomes more indirect, and that sensitivity is skewed toward stereotype-consistent readings. We call the ratio of this skew on indirect versus direct forms the pragmatic amplification ratio. Second, specializing a compact model on literal domain data raises this ratio, because specialization removes the general conversational competence the model needed to resolve the request without leaning on priors. Third, a consistency objective that anchors the indirect reading to the model's own reading of the explicit paraphrase reduces the ratio more than augmentation plus demographic swapping does. It does so without erasing attributes that are legitimately relevant, and without making the model over-interpret literal inputs.

The novelty is in the second and third claims. The closest work stops one step short of each. Indirect-request work measures accuracy but never varies who is speaking (Making Task-Oriented Dialogue Datasets More Natural…). Underspecified-question work measures stereotyping when the context gives the model nothing to go on, not when the information is present but implicit (UnQovering Stereotyping Biases via Underspecified Questions). Distillation-time debiasing equalizes single-token pronoun probabilities in decontextualized templates (FairDistillation: Mitigating Stereotyping in Language Models). Work on "pragmatic hallucination" (reading subtext into text that should be taken literally) links that habit to a contextual bias but never to social groups (Born Pragmatic, Trained to Hallucinate?).

Research Program

Benchmark. We build a paired benchmark by extending the generation recipe behind INDIRECTREQUESTS. That recipe starts from an explicit intent in the Schema-Guided Dialogue corpus and rewrites it with five strategies: justification, elaboration, hyponym swap, synonym swap, and small talk. We add a second domain closer to the allocational harms in the bias literature: a recruiting and clinical-intake assistant that maps requests to structured actions. Examples of such actions are which role to draft an application for, or which triage queue applies.

Each item has a fixed gold intent, one direct form, several indirect forms, and a short speaker persona with a single varying attribute. Annotators also mark a stereotype-consistent distractor, an incorrect reading that a stereotype would make plausible. For example, "I'd like something less demanding" might be misread as a junior role for one persona and as part-time hours for another.

Two item types serve as controls. In attribute-relevant items the persona attribute is stated as mattering, for instance age for a pediatric queue; here a correct model should change its answer. In literal items no inference is warranted; these follow the context-flip logic of PaCE. We borrow the permutation controls of UnQovering Stereotyping Biases via Underspecified Questions, swapping persona order and attribute position, so that positional artifacts are not counted as bias.

We score implicitness in two ways: by the discrete rewriting strategy, and by ImpScore, a learned measure of how far a sentence's intended meaning sits from its literal meaning. ImpScore is reported to give unreliable scores on technical jargon, so in the clinical domain we treat it as secondary.

Measurement. For item $i$, let $b_i$ be the difference across persona pairs in the probability of choosing the stereotype-consistent distractor, after permutation correction. The pragmatic amplification ratio is the mean of $b_i$ on indirect forms divided by its mean on direct forms. We compute this ratio and plot it against implicitness level for three groups of models.

The first group is compact base models: Flan-T5-xl and xxl, and a 7–8B open model. The second is the same models specialized in three ways: supervised fine-tuning on direct SGD dialogues, chain-of-thought distillation from a large teacher in the style of Specializing Smaller Language Models towards Multi-Step Reasoning, and preference tuning in the style of Rethinking Pragmatics in Large Language Models. The third is a large generalist, Llama 2 70B.

That specialization paper documents a severe loss of general reasoning, which it calls a "specialization tax." Our second claim predicts a corresponding social tax on top of it.

Training fix. Let $x^{\text{lit}}$ be the explicit paraphrase, $x^{\text{ind}}$ an indirect rewrite with the same gold intent $y$, and $a$ the persona. We train with

$$ \mathcal{L} = -\log p_\theta(y \mid x^{\text{ind}}, a) + \lambda \, \mathrm{KL}\big(\,\mathrm{sg}[p_\theta(\cdot \mid x^{\text{lit}}, a)] \;\|\; p_\theta(\cdot \mid x^{\text{ind}}, a)\big), $$

where $\mathrm{sg}$ means the anchor distribution is held fixed rather than trained through, and $\lambda$ weights the consistency term. The anchor uses the same persona. That is why the fix differs from forcing invariance across personas. Where the explicit request genuinely depends on the attribute, that dependence is kept. Where it does not, the indirect reading inherits the direct form's attribute-insensitivity.

A variant replaces the self-anchor with a teacher's reading of $x^{\text{lit}}$. The objective has an obvious failure mode: if direct readings are themselves biased, anchoring copies that bias into the indirect readings. The measurement stage tests whether this happens before we rely on the fix.

Baselines

The do-nothing reference is the compact model fine-tuned on direct SGD dialogues only, which is the standard specialization pipeline. Its reported analogue is the T5 tracker at 0.13 accuracy on INDIRECTREQUESTS, down from 0.51 on the standard test (Making Task-Oriented Dialogue Datasets More Natural…). We re-run it on our persona-augmented split to obtain its amplification ratio. No reported result exists on Lacuna for demographic bias on indirect-request dialogue state tracking.

The strongest reported system on the accuracy side is Llama 2 70B, prompted without fine-tuning, at 0.69 accuracy on INDIRECTREQUESTS according to the same paper. It shows what scale buys. To match the 70B on accuracy, a specialized compact model must exceed 0.69 on the original INDIRECTREQUESTS items. To beat it on fairness, it must show a lower amplification ratio on our split.

The strongest competing training method is indirect-request augmentation combined with counterfactual persona swapping. This means supervised fine-tuning on generated indirect requests with gold intents, where each example is duplicated across persona variants. It addresses both failures with standard tools, so it is the real threat. It receives exactly the same generated data, the same number of gradient steps, and the same hyperparameter search budget of 20 trials per method on a held-out dev split, as our method.

We also run a teacher-side equalization baseline adapted from FairDistillation. It equalizes the teacher's output distribution across persona variants before distillation, extending that paper's pronoun rule to whole intent labels.

Two ablations isolate the key ingredient. The first keeps the consistency loss but removes persona variation from training, which tests whether the anchor alone transfers fairness. The second replaces the same-persona literal anchor with a persona-averaged anchor, which turns the method into invariance enforcement and should hurt the attribute-relevant items.

Beating the strongest competitor requires three results together. The amplification ratio must be significantly lower (bootstrap over items). Accuracy on indirect items must be within noise of the competitor or better. Accuracy on attribute-relevant items must not degrade.

Why not just…?

Why not just use a large general model? Scale narrows the accuracy gap but does not close it (0.69 versus 0.96 on direct requests). Bias evidence also points the other way: stereotyping intensity grows with capacity in UnQovering, and hurtful completions sit at the peak of larger models' predicted distributions in FairBelief. Our measurement stage includes Llama 2 70B. If its amplification ratio is near 1, the scaling argument wins on fairness, and the case for our method rests only on deployment cost.

Why not just fine-tune on indirect requests with demographic swaps? This is the most serious alternative, and it may well match us on the amplification ratio alone. Swapping personas enforces invariance even where the attribute should matter, so we expect it to lose on attribute-relevant items. That expectation is a hypothesis, and the comparison above settles it. If swap augmentation matches us on every split, including the relevant-attribute items, we will report that the fix is unnecessary and that the measurement is the contribution.

Why not just prompt with chain-of-thought or a "be literal" instruction? Chain-of-thought narrows but does not close the context-sensitivity gap in PaCE (−18.1% to −9.7%). A strict-literal prompt overshoots in the opposite direction (GPT-4o from −10.4% to +26.1%), suppressing inferences that are actually needed, which would destroy accuracy on indirect requests. We run both prompts on every model as inference-time baselines.

Why not just ask a clarifying question? In deployment, clarification is the right behavior for genuinely ambiguous requests. Our items, however, are not ambiguous to humans: the intent is recoverable. A model that asks for clarification more often for some personas than others is itself showing bias. We therefore score per-persona clarification rate as a secondary outcome rather than treating clarification as a remedy.

Decisive Checks

The program succeeds only if four things hold. The amplification ratio must exceed 1 and rise with implicitness level for compact models. Specialization on direct data must raise it relative to the unspecialized base. The consistency objective must lower it below the swap-augmentation competitor while keeping attribute-relevant accuracy intact. Finally, the fix must not trade one failure for another. We check this on PaCE, whose average context-sensitivity gap across models is reported as −14.0% (Born Pragmatic, Trained to Hallucinate?); a more negative gap after training would mean the model over-reads literal contexts. We also report PUB (PUB) and BBH accuracy to show whether the specialization tax worsens.

If the first check fails, so that indirectness does not amplify bias, the whole premise is refuted. That negative result would still be worth publishing, because it would mean template-based bias audits are not missing the dominant failure.