Large language models know a great deal about the world yet do poorly on benchmarks about everyday social norms, such as when an apology is owed or how close to stand to a stranger. On these benchmarks they lose to small fine-tuned classifiers and give different answers when a prompt is reworded. This note proposes a controlled test of one specific explanation for part of that failure. In these benchmarks, a model must sort a situation into a category whose name is a terse, abstract word like "criticism," "request," or "proxemics." A fine-tuned classifier learns from thousands of labeled examples where the annotators drew the line between those categories. A prompted generalist model sees only the word. If that is a large part of the gap, then replacing each terse label with a behavioral specification should help. Such a specification would state what counts, what does not, and how the category differs from its nearest neighbor. It should raise accuracy, make predictions more consistent across rewordings of the prompt, and close much of the distance to the fine-tuned model, without any weight updates.
The test matters because three research threads currently read the same symptoms in different ways. Work on the normative reasoning gap treats low scores as evidence that models cannot reason about social rules. Work on prompt sensitivity treats unstable answers as a property of the model's decoding. Work on generalist models finds that large general models beat domain-tuned ones in medicine and chemistry, so it is puzzling that the reverse appears for social norms. One manipulation, enriching what the category labels mean while keeping the inputs fixed, can separate these readings. If enrichment helps on tasks where labels carry the decision but not on tasks where the decision rests on choosing an action, we learn where the reasoning gap actually lives.
Core Claim
We claim that on label-centric normative tasks, a measurable share of generalist model error and prompt brittleness comes from not knowing where the category boundaries are, rather than from faulty reasoning about norms. We further claim this share can be removed with contrastive boundary specifications. These are short label descriptions that say what the category includes, what it excludes (including neutral context-setting), and how it differs from the most confusable other category. The claim is falsifiable in three ways. First, contrastive specifications should beat plain definitions of equal length. If they do not, the effect is generic extra context, not boundary information. Second, the gain should shrink on a task where answers are actions rather than category names. If it does not shrink, we are measuring something other than label semantics. Third, sharper output distributions should come with better calibration, not just more confident wrong answers.
The motivating evidence comes from several sources. In vision-language few-shot classification, Exploiting Category Names for Few-Shot Classification with Vision-Language Models initializes a classifier from text embeddings of class names. This lifts EuroSAT 5-shot accuracy to 85.78% from a 49.74% zero-shot baseline, and the gain collapses when names are replaced with opaque IDs like "Class 001." That shows label semantics can carry most of a few-shot gain, but only for concrete, visually grounded objects. ShefCDTeam at SemEval-2024 Task 4 finds text-to-text models degrade on abstract rhetorical labels that lack lexical markers in the input. Normative categories are exactly of that kind. ProSA links prompt sensitivity to low decoding confidence: when the model is unsure, small rewordings tip the answer. ProSA never tested normative tasks or asked whether label clarity drives that low confidence. RENOVI supplies the puzzle. A fine-tuned RoBERTa-zh reaches 60.56 F1 on norm-category prediction, while GPT-3.5 struggles at zero-shot detection yet writes remediations judged effective 87% of the time. A model that can repair a violation but cannot name its category is a natural suspect for a label-boundary problem.
The constraining evidence comes from EgoNormia. It attributes over 70% of multimodal model errors to normative reasoning and under 25% to perception, and humans score 92.4% against 53.9% for Gemini 2.5 Pro. Its questions ask for the most appropriate next action, so the answer is not a category name. We therefore predict label enrichment will do little there, and we use EgoNormia as a negative control rather than a target.
Research Program
The central artifact is a label-semantics ladder applied to fixed inputs. Each normative category is presented at one of four levels while the input situations and answer format stay identical:
- An opaque ID ("Category C").
- The benchmark's original terse name.
- A plain definition.
- A contrastive boundary specification.
The plain definition and the contrastive specification are matched in token length, so any difference between them reflects content rather than amount of text.
Contrastive specifications are written in two ways so that the comparison with fine-tuned models stays honest:
- Theory-derived. Written once from each benchmark's own annotation theory, for example the Expectancy Violation framing in RENOVI. This version uses no labeled training data.
- Data-induced. An LLM reads the training split's confusion pairs and drafts boundary statements, which are frozen before any test-set evaluation. This version uses the same labeled data a fine-tuned model sees, so any advantage over fine-tuning cannot be blamed on extra information.
Prompt sensitivity. For each model, ladder level, and instance, we measure sensitivity across 12 instruction templates in ProSA's four functional groups, plus random permutations of answer-option order and label symbols. The permutation perturbations follow When Benchmarks are Targets, which shows that option shuffles and symbol swaps alone can move leaderboard rankings by up to eight places. Our instance-level sensitivity is
$$s(x) = 1 - \max_{c} \frac{1}{P}\sum_{p=1}^{P} \mathbb{1}[\hat{y}_p(x) = c],$$
where $x$ is an instance, $P$ is the number of perturbed prompts, $\hat{y}_p(x)$ is the model's predicted category under perturbation $p$, and $c$ ranges over categories. A score of 0 means every perturbed prompt gives the same answer.
Other measurements. We record the probability the model assigns to its top answer, to test ProSA's confidence account directly. We also record the logit-text mismatch: how often the answer read off next-token probabilities disagrees with the category the model states in a free-text justification. The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models reports this mismatch exceeding 50% for Llama-2 on value questions, but does not test any intervention against it.
Fine-tuning question. We ask whether the generalist superiority pattern reasserts itself once labels are clear. We compare prompted generalists at each ladder level against a reproduced RoBERTa-zh and a LoRA-tuned 7B Qwen model trained on RENOVI. We then test both models on NormDial, a bilingual adherence/violation dataset (NormDial). This checks whether narrow tuning buys in-domain F1 at the cost of transfer, the kind of hidden cost MedREQAL and FGBench document in medicine and chemistry.
Models.
- Text tasks: Qwen2.5-72B-Instruct and Llama 3-70B-Instruct as open generalists, chosen in part because ProSA finds the latter unusually stable, which makes it a useful contrast. One closed model is added where budget allows.
- EgoNormia: Qwen2.5-VL-72B.
- Second normative text task: the Contextual Integrity synthetic set (Contextual Integrity Synthetic Dataset), 729 examples, where the labels are information-flow norms and should behave like RENOVI.
Baselines
The naive reference point is the benchmark's terse labels with a single standard prompt and zero examples. This is how these models are usually scored. The opaque-ID rung sits below it as a floor that tells us how much the label words already contribute. Following the category-names paper, we expect the opaque-ID rung to fall well below the terse rung on RENOVI.
The strongest existing method for in-domain accuracy is supervised fine-tuning. Concretely, this is RoBERTa-zh on RENOVI, which RENOVI reports at 60.56 F1 on category prediction and 78.33 F1 on impact estimation. We reproduce it on the same splits. We also add a LoRA-tuned 7B Qwen model with the same training data and a hyperparameter budget of 20 configurations. This matches the 20 specification drafts the data-induced condition may try on the development split. Beating fine-tuning means a prompted generalist with data-induced specifications meets or exceeds the reproduced RoBERTa-zh macro-F1 on the human-authored test dialogues, within a bootstrap confidence interval. It must do so while showing lower sensitivity $s(x)$ than the same generalist with terse labels.
For selective prediction, meaning abstaining on low-trust answers, the strongest method is SENSEL from On the Relation between Sensitivity and Accuracy in In-Context Learning. SENSEL abstains when predictions disagree across perturbed prompts. It reports +4.1 points of area under the F1-coverage curve over maximum softmax probability (MaxProb) on GPT-J-6B, and +3.0 using demonstration ordering alone. We run SENSEL and MaxProb with identical perturbation sets at each ladder level. We also apply Prototypical Calibration, a correction for the model's built-in preference for some label tokens, because that paper shows label-token bias can mask true sensitivity. The question is whether rich specifications let single-pass MaxProb approach multi-pass SENSEL, which would recover most of the reliability at a fraction of the compute.
The ablation that isolates our key ingredient is the length-matched plain definition. If contrastive specifications do not beat it, boundary information is not doing the work. A second ablation, the shuffled specification, attaches each category's specification to the wrong category. It rules out the possibility that any extra descriptive text helps regardless of what it says.
Why not just…?
Why not just add one or two demonstrations? This is the most serious threat. ProSA finds the move from zero-shot to one-shot gives the strongest stabilization of any intervention it tests. We address it with a direct comparison. We test k-shot demonstrations at token budgets matched to the specifications, specifications alone, and both together, on the same instances. Two pieces of evidence suggest demonstrations may not fully substitute. Demonstration order itself creates sensitivity, which SENSEL exploits for +3.0 AUC. And the AOV survey reports that examples shift measured values in normative settings. If demonstrations match specifications on accuracy and sensitivity, we will report that the boundary information is equally well carried by examples. The deliverable then becomes a diagnosis of the gap rather than a new prompting method, and the EgoNormia contrast still stands.
Why not just fine-tune on norm data? On the in-domain test it may well win, and our baseline section treats it as the bar to clear. The case against relying on it is the transfer cost seen elsewhere: agreeability bias after narrow tuning in MedREQAL, and RENOVI's own report that training on synthetic dialogues degrades under domain shift. The NormDial transfer experiment settles whether that cost appears for norms.
Why not just vote over many prompts? Majority voting or SENSEL-style abstention reduces brittleness, but it multiplies inference cost by the number of variants, a limitation the file attributes to SENSEL. We do not claim specifications make voting useless. We measure how much of SENSEL's selective-prediction gain survives after specifications are added. A small residual would mean label clarity removes most of what voting was compensating for.
Why not just prepend a coarse category tag? Wang et al. show that tags like "Social Norms" help considerably. The terse-label rung essentially is that condition. Li et al. report that models still flag neutral context-setting as violations, an error a tag cannot address but an explicit exclusion clause can. We measure this directly as the false-violation rate on RENOVI and NormDial dialogues that contain no violation.
Decisive Checks
Five results would confirm the account:
- On RENOVI category detection, accuracy rises from opaque ID to terse name to plain definition to contrastive specification, with the last step significant at matched length.
- Mean instance sensitivity $s(x)$ falls along the same ladder.
- Logit-text mismatch drops, measured by comparing free-text justifications against logit answers.
- On EgoNormia's action-selection questions, the gain from specifications applied to its seven norm taxonomies is near zero. This would confirm that EgoNormia's over-70% reasoning errors are not a label artifact.
- Expected calibration error does not worsen as confidence sharpens.
Three results would falsify or narrow the account. A uniform gain across RENOVI and EgoNormia would mean generic context, not boundary information, drives the effect. Confidence sharpening without accuracy gains would mean specifications create false certainty. Matched demonstrations equaling specifications would reduce the claim to a diagnosis rather than a method. Any of these outcomes still answers the question that motivates the project: whether the normative reasoning gap measured on current benchmarks is about norms, or partly about category names.
References
- Exploiting Category Names for Few-Shot Classification with Vision-Language Models
- ShefCDTeam at SemEval-2024 Task 4: A Text-to-Text Model for Multi-Label Classification
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversations
- EgoNormia: Benchmarking Physical-Social Norm Understanding
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
- The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models
- NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violation
- MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering
- FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- Semantic Categorization of Social Knowledge for Commonsense Question Answering