Where Norm Judgments Break: Stage-Localized Prompt Sensitivity for Targeted Human Repair

Large language models can explain quantum mechanics but still misjudge whether a line of dialogue breaks a social norm. On NormDial, a bilingual dataset of dialogues that either follow or break cultural norms, ChatGPT reaches roughly 0.81–0.83 F1 on turns that follow a norm but only 0.66–0.68 on turns that break one, and it often flags neutral scene-setting turns as violations (NormDial). We propose to find out where in the model's reasoning these judgments break. The method has three parts. We make the model reason in short, structured stages: what the turn is doing, who is speaking to whom, which norm applies, and the verdict. We then reword the prompt in ways that should not matter, such as reordering options, paraphrasing the instruction, or adding a trailing "thank you". We record which stage's answer flips first. Human annotators are sent only to that stage, they correct one short slot, and the corrected chains become training data.

The bet is specific. Most label flips in norm-violation detection begin in the early stage where the model reads what a turn is doing, not in the stage where it applies a norm. If that holds, prompt sensitivity is more than an evaluation nuisance. It acts as a pointer that tells a human reviewer which sentence of the reasoning to fix. That would make human-in-the-loop data creation for social norms cheaper per corrected label and less brittle than relabeling final answers.

Core Claim

Prompt sensitivity, meaning how much a model's output changes under rewordings that should not matter, is known to track error. Chen et al. find a negative correlation between sensitivity and accuracy reaching Pearson $r=-0.40$ and use sensitivity to decide when to abstain (On the Relation between Sensitivity and Accuracy in In-Context Learning). ProSA ties it to flat, low-confidence output distributions (ProSA). Both treat the model's answer as a single block.

Chain-of-thought prompting, which has the model write intermediate reasoning before answering, splits that block into steps. Cai et al. show that up to half of chain-of-thought failures trace to one faulty intermediate step, and that humans fix a flagged step about twice as fast as they solve from scratch (Human-in-the-Loop through Chain-of-Thought).

Our claim joins these results. When the chain is split into stages with short categorical outputs, sensitivity can be measured per stage. The earliest unstable stage then marks the error well enough that a human edit to that stage alone repairs the final judgment. For norm detection in particular, we predict that this stage is usually the turn-function reading. That reading is where the documented confusion of scene-setting with violation would live, and where explicit relationship context helps so much (SocialDial).

Research Program

Stage-structured reasoning. The reasoning scaffold borrows from Cognitive Chain-of-Thought, which orders social reasoning as perception, then situation, then norm. That ordering improved GPT-4o by 8.0 points and Gemini 1.5 Pro by 14.1 points over standard chain-of-thought on VAGUE (Cognitive Chain-of-Thought). We adapt the scaffold to dialogue. Stage 1 names the turn's function (for example scene-setting, request, criticism, apology). Stage 2 states the speaker–listener relationship and status. Stage 3 names the applicable norm. Stage 4 gives the adhere/violate/not-relevant verdict. Each stage must emit a short categorical slot followed by a sentence of justification. The categorical slot lets us compare stages across prompt variants without an LLM judge, which would itself be brittle.

Per-stage instability. For each instance we run $K$ meaning-preserving prompt variants. Following the perturbation families documented in When Benchmarks are Targets and The Butterfly Effect of Altering Prompts, these are option and label reordering, instruction paraphrase, whitespace and sign-off changes, and same-culture name swaps. A human spot-check confirms that a sample of variants preserves meaning. The instability of stage $j$ is

$$s_j = 1 - \max_c \frac{1}{K}\sum_{k=1}^{K} \mathbb{1}[y_j^{(k)} = c],$$

where $y_j^{(k)}$ is the slot value at stage $j$ under variant $k$ and $c$ ranges over that stage's categories. The flagged stage is the earliest $j$ with $s_j$ above a threshold tuned on a development split. We count a flag as causal only when the downstream verdicts change along with that stage across variants.

Targeted human repair. Annotators see the chain with the flagged stage highlighted. They correct only that slot and its sentence. The model then regenerates the later stages conditioned on the fix. The corrected chains, together with their perturbed prompts, become fine-tuning data for an open 7B model. Contextual Integrity work showed that a few hundred well-targeted examples can make a 7B model beat a 14B one (Contextual Integrity in LLMs via Reasoning and Reinforcement Learning). That work's data was purely synthetic, and it never tested whether rewording breaks the learned behavior. We test both.

Countering rubber-stamping. Reviewers in long verification pipelines tend to pass errors without reading closely. To measure this, we seed queues with known-wrong chains and report how often annotators catch them.

Baselines

The naive reference point is direct zero-shot classification with a single fixed prompt. For it we have reported numbers to beat: NormDial violation F1 of 0.66 (Chinese) and 0.68 (English) for ChatGPT (NormDial), and on ReNoVi, 60.56 F1 on category prediction and 78.33 on impact estimation for fine-tuned RoBERTa-zh (RENOVI). We rerun the zero-shot baseline with our current models and perturbation set rather than trusting the old figures, because those figures came from a single prompt.

The strongest existing method for routing human effort into reasoning chains is the Manual Correction System. It samples several rationales, flags instances whose answers disagree, measured as Diversity Entropy, and lets humans edit any step. It raised SVAMP accuracy from 68.00% to 80.10% when combined with self-consistency (Human-in-the-Loop through Chain-of-Thought). We found no reported result on Lacuna for this method on any norm or social benchmark. We run it with the same model, the same stage scaffold, the same number of forward passes per instance ($K$ samples versus $K$ perturbed prompts), and the same total annotator minutes. Beating it means more violation-F1 gain per annotator hour, and lower flip rates after fine-tuning, both on held-out perturbation families.

A second competitor is relabeling final labels: annotators see the dialogue and the model's verdict and fix only the verdict, on instances chosen at random or by sentence-level sensitivity (sensitivity-based selective prediction, SENSEL), again within the same time budget. Two ablations isolate our key ingredients. The first keeps perturbation-based flagging but hides which stage is unstable, which isolates localization. The second computes per-stage instability from temperature sampling instead of prompt perturbation, which isolates the perturbation signal itself. As a prompting-only reference we include standard "think step by step" prompting, the unedited stage scaffold, and one-shot anchoring, which ProSA found gives the largest single robustness gain.

Why not just…?

Why not just use a bigger model? On EgoNormia the best frontier model reaches 53.9% against 92.4% for humans, and over 70% of its errors are normative rather than perceptual (EgoNormia). ProSA reports that GPT-4o can be more prompt-sensitive than Llama 3-70B-Instruct. Scale alone does not remove the problem. We still evaluate one frontier model with the full perturbation set, so the size of its residual brittleness on norm tasks is measured rather than assumed.

Why not just use Diversity Entropy from sampling, which is already proven? This is the real threat. Sampling disagreement may flag the same instances as prompt perturbation, and more cheaply. Our sampling-based per-stage ablation settles the question directly. If it localizes errors as well as perturbation does, we report that the contribution is stage localization and not the perturbation signal, and we adopt sampling. Our prior that perturbation adds something rests on one reason. Sampling measures how spread out the model's output distribution is under a fixed prompt. Perturbation probes whether the model's reading of the situation is anchored in content or in surface form, and the turn-function confusion described in NormDial looks like the latter.

Why not just have humans relabel final answers? That is cheaper per item and needs no scaffold. But a corrected label teaches nothing about why, and RENOVI shows that models trained on synthetic dialogue degrade under domain shift (RENOVI). Our hypothesis is that corrected turn-function readings generalize better than corrected verdicts. The matched-time comparison and the held-out test on ReNoVi's human-authored dialogues will show whether they do. If final-label relabeling matches us on both, the stage editing is not worth its cost.

Why not just give the model the relevant norm? Bian et al. found that injecting correct commonsense knowledge did not reliably help and even lowered Social IQA accuracy, with ChatGPT recalling relevant knowledge at 84.4% but with only 55.8% precision (ChatGPT Is a Knowledgeable but Inexperienced Solver). NormDial dialogues are each built from a known norm, so we can test this alternative exactly by handing the model the gold norm. Low violation F1 and frequent flips that persist with the gold norm in hand would be evidence that the failure sits upstream, in reading the situation.

Proposed Experiments

The main testbed is NormDial, with its human-verified turns as the test set, since its labels were themselves produced by chain-of-thought. ReNoVi supplies a second culture and task. Its 512 human-authored dialogues serve as the out-of-distribution test that probes domain shift. Social IQA, where ChatGPT scores 62.0% (ChatGPT Is a Knowledgeable but Inexperienced Solver), and the Contextual Integrity vignettes (ContextualIntegritySyntheticDataset) test whether reduced brittleness transfers to other social reasoning. We report violation F1, the flip rate under held-out perturbation families, F1 gained per annotator hour, and the catch rate on seeded errors.

The first decisive check is diagnostic and needs no humans in the loop. On about 500 turns hand-annotated for turn function, we measure where flips originate, and we run oracle interventions that replace one stage's slot with the gold value. The localization claim fails if flips are spread evenly across stages, or concentrate in the norm stage. It also fails if inserting the gold turn function recovers no more F1 than inserting the gold norm.

The second check is the matched-budget repair comparison against the Manual Correction System, final-label relabeling, and the two ablations. The third is fine-tuning on each condition's corrected data, evaluated on ReNoVi's human-authored subset under unseen perturbations. That test asks whether stage-corrected chains teach a reading of social situations that survives rewording, or only another surface pattern.