Language models can often name the emotion in a sentence. It is much less clear whether they work it out from the situation. Psychologists have long held that emotions follow from how a person judges an event: whether it helped or blocked their goals, who was responsible, and whether it was fair. Anger is a blocked goal blamed on someone else; regret is the same blocked goal blamed on oneself. Recent tests show that models skip this reasoning. On a Prisoner's Dilemma benchmark, they answer "anger" almost whenever unfairness is mentioned and confuse self-blame with other-blame. Those tests use a few hundred fixed, hand-written stories, and fixed stories have two problems. They cannot separate "the model cannot reason" from "the model saw similar text during training." They also cannot vary difficulty one factor at a time.
We propose a generator that builds each emotion story from a small hidden graph. The graph lists the events, who caused them, and the chain of inferences a reader needs to reach each judgment. A rule table derived from appraisal theory supplies the correct emotion, and human raters check that the table matches how people read the stories. Because stories are generated fresh, they cannot have been memorized. Because the graph is known, we can raise inference depth, add distractors, or build an easy twin of every hard item. On this test bed we ask one main question, under a fixed and equal tuning budget for every method. Do the emotion-reasoning failures come from the model failing to extract judgments such as blame, or from the model combining them with shortcuts? And does splitting the task into isolated sub-questions fix either problem?
Core Claim
We claim two things. First, the "fairness means anger" shortcut and the confusion between self-blame and other-blame are structural failures, not memorization artifacts. If so, they should get worse in a measurable, predictable way as the number of inference steps needed to establish responsibility grows. Second, much of the failure sits in the combining step rather than the extraction step. Asking the model for each judgment in a separate, clean context, and then combining the answers, should cut shortcut errors sharply. Most of that gain should survive whether the final combination is done by the model or by a fixed rule. If the claim is wrong, the experiments will show it cleanly: accuracy would stay flat across depth, or decomposition would help only when a rule does the combining.
Relation to Prior Work
Three papers motivate the proposal directly. Beyond Context to Cognitive Appraisal introduced ToMEmoReason, which has 432 forward and 150 backward Prisoner's Dilemma vignettes. It found forward accuracies between 52.0% (Llama 3.1 8B) and 57.9% (Gemma 7B), against 16.7% chance, along with the fairness-dominance and accountability-blindness patterns. Its items are fixed and cover three domains, and the difficulty of each item is not controlled. We keep its theory and its failure categories. We replace the fixed items with generated items whose difficulty we control.
DyVal supplies the method. It generates reasoning problems from graphs of dependent steps (directed acyclic graphs) and shows two things: models strong on fixed benchmarks drop to near zero on fresh graphs, and depth hurts more than width. DyVal is limited to arithmetic, formal logic, and graph search. We carry its generation idea into natural-language social judgments, where the correct answer comes from a psychological theory rather than a calculator. That change forces a human-validation step DyVal never needed.
Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones? supplies the consistency measure. Its consistency score is the probability of solving the easy version given that the hard version was solved. It reports 92.2% for GPT-4 Turbo and 50–70% for 7B models on items derived from GSM8K, HumanEval, and IFEval. We apply the same measure to easy/hard twins that our generator produces by construction.
On the method side, When Splitting Makes Stronger proves two results about divide-and-conquer prompting, which solves sub-questions in separate contexts. Its expressive power is a proper subset of chain-of-thought, meaning step-by-step reasoning in a single context. It also limits how errors spread between steps. Empirically it reaches 76.92 F1 on long multiplication, against 69.41 for Tree-of-Thoughts. This produces a tension we test directly. Judgments about an event interact with one another, so isolating them could hurt by cutting those interactions, or help by blocking shortcuts.
SPECS already structures emotion-cause extraction as a sequence of appraisal steps with repeated self-consistency passes. It reports relative F1 gains, for example +4.78% on emotion-cause pair extraction, but no absolute values. It runs on fixed Chinese corpora and does not separate extraction errors from combination errors. That separation is our central ablation.
E-ICL improves fine-grained emotion recognition by more than 4% on GoEmotions and other datasets. It does this by retrieving examples that are emotionally similar, chosen with an auxiliary RoBERTa-large model, and by excluding implausible labels. It is our strongest example-selection baseline.
One paper constrains our novelty claim. A Cognitive Battery for Foundation Models already generates social-cognition items procedurally, 25,390 items across 1,138 sub-types. It is not organized around appraisal variables, does not pair easy and hard items, and does not target fine-grained emotion labels. We therefore claim neither to be first to procedurally generate social items nor to have invented a new benchmark genre. We claim only the appraisal-graph structure and the extraction-versus-combination diagnosis it enables.
Research Program
The generator. Each item starts from a graph with three layers. The first layer holds events, each with actors and intentions. The second holds five appraisal variables, meaning the judgments a person makes about an event: goal conduciveness, agency/accountability, fairness, certainty, and expectedness. The third holds an emotion, taken from the twelve emotion types used in SPECS under the Ortony–Clore–Collins model: gratitude, remorse, reproach, relief, disappointment, pride, shame, admiration, regret, anger, hope, and fear.
The generator exposes three difficulty controls:
- Depth is the number of stated facts that must be chained to establish an appraisal. For example, accountability at depth 3 requires learning who changed the plan, that they knew the consequence, and that the protagonist had no say.
- Width is the number of agents and events.
- Distractors are surface cues that conflict with the correct appraisal. An example is the word "unfair" applied to a background event that did not affect the protagonist.
Each hard item comes with an easy twin that states the same appraisals directly. Each item also comes with a counterfactual partner that flips exactly one appraisal, such as moving blame from another person to the self, so that the correct emotion flips from anger to regret. The stories are written by filling templates across many domains beyond Prisoner's Dilemma, such as workplaces, families, sports, and health. A light rewording pass then adds variety to the text.
We also build a structurally matched regeneration of ToMEmoReason's own configurations. Accuracy on our versions can then be compared directly with accuracy on the original items.
Label validity. Labels computed from a theory measure whether a model agrees with that theory, not whether it understands emotions. So before any model is evaluated, three raters label 600 generated items drawn across all templates and depths. We keep only templates on which the rater majority agrees with the rule label at a pre-set rate. We report this agreement next to every model result. Templates people disagree on are dropped, not argued for.
The method under test. Appraisal decomposition asks the model for each of the five appraisals in a separate, fresh context. Each sub-question contains only the story and one appraisal question. The five answers are then combined. We compare two ways of combining them. In the first, the model combines them in a final prompt. In the second, the fixed rule table combines them.
Two additional variants isolate the ingredients. One asks all five appraisal questions in a single shared context, so decomposition is present without context isolation. The other ends with an "oracle" step that feeds the true appraisals to the model's own combination step, which shows the ceiling reachable if extraction were perfect.
This design splits every error into two parts: errors in extracting the appraisals, and errors in combining them. It is the diagnosis that neither ToMEmoReason nor SPECS provides.
Equal tuning under a budget. Every method gets the same small search: 16 prompt configurations, sampled at random over temperature, number of examples, and example order. The search is scored on a development set generated from one random seed. We report results on a test set from a different seed.
We report two numbers for every method: the best configuration's score, and the spread across all 16 configurations. Decomposition makes more model calls than single-prompt methods, so we also match compute. Every method is compared against chain-of-thought with majority voting, given the same total number of generated tokens. This addresses the confound noted in Jargon, where a new method is tuned carefully while baselines run on defaults.
Baselines
Direct zero-shot prompting. The model reads the story and names the emotion, with no reasoning prompt. This is the do-nothing reference and matches ToMEmoReason's setting. The strongest reported number on that benchmark is 57.9% forward accuracy for Gemma 7B, from Beyond Context to Cognitive Appraisal. Our first sanity check is that direct prompting on our structurally matched regeneration falls near the reported 52–58% range.
Chain-of-thought and majority voting. Chain-of-thought prompting ("think step by step") and chain-of-thought with majority voting over sampled answers are the strongest cheap competitors. Majority voting is run at the same token budget as decomposition, so any gain from decomposition cannot be explained by extra compute. Decomposition beats this baseline only if two conditions hold. Its median configuration must exceed voting's best configuration in accuracy at depth 3 and above. It must also cut the shortcut rate on counterfactual pairs, where "shortcut rate" means answering anger when blame falls on the self.
E-ICL. This is E-ICL, with its auxiliary RoBERTa-large model trained on a disjoint set of generated templates. It is the strongest example-selection method in the file. No absolute numbers are reported for it; the file gives only gains "over 4%", and no reported result was found on Lacuna for its per-dataset GoEmotions F1. It is the right competitor for the view that better examples, not better reasoning structure, are what matter.
SPECS-style prompting. This is multi-pass appraisal prompting with self-consistency, following SPECS, adapted to our label set. It is the closest existing structured method. No absolute F1 is reported for it in the file, so we compare it only on our own data.
Ablations of decomposition. The shared-context variant separates the effect of context isolation from the effect of decomposition. The rule-combination and oracle-appraisal variants separate extraction errors from combination errors.
Why not just…?
Why not just reword ToMEmoReason's vignettes to remove any memorization? Rewording is cheap and would detect memorization of wording. We include it as a condition. It cannot change difficulty one factor at a time or produce matched easy/hard twins, and those are what the depth and consistency analyses need.
We also accept a possibility that would weaken one of our motivations. The reported accuracies are already low, so contamination may not be inflating them much. If accuracy is the same on original, reworded, and regenerated items, we will report that the failures on this benchmark are not contamination artifacts. The generator would then be justified only by the depth and consistency measurements it makes possible.
Why not just use a stronger reasoning model? ToMEmoReason already reports o3-mini at 55.6%, below Gemma 7B at 57.9%. This suggests that formal reasoning ability does not automatically carry over to emotion reasoning. We include o3-mini to test whether its disadvantage grows or shrinks with inference depth. If it shrinks, the problem is more about social cues than about reasoning. If it grows, the problem is more about the reasoning steps themselves.
Why not just fine-tune a small model on emotion data? Fine-tuned BERT reaches only 0.46 average F1 on GoEmotions' 27 classes, as reported in GoEmotions. A more serious threat is fine-tuning directly on our generated items. We run a LoRA fine-tune of Llama 3.1 8B on items from half the templates at depths 1–2. LoRA trains small add-on weight matrices instead of the full model. We then test the fine-tuned model on held-out templates and on depths 3–4. If it matches decomposition there, the failures can be fixed by training, and the benchmark risks rewarding template learning. We would report that as the main finding.
Why not just use the Cognitive Battery's social items? Those items cannot produce counterfactual partners that flip one appraisal, and they cannot split errors into extraction versus combination. Those two capabilities are what our questions require.
Proposed Experiments and Decisive Checks
Models. We evaluate Gemma 7B, Mistral 7B, and Llama 3.1 8B, matching ToMEmoReason, plus o3-mini. Each is run on 500 generated test items for each cell of a grid crossing depth (four levels) with distractors (present or absent), under the equal 16-configuration budget.
Metrics.
- Accuracy over the twelve emotion types.
- Consistency score on easy/hard twins, as defined by Yang et al.
- Shortcut rate on counterfactual pairs.
- The split of each method's errors into extraction and combination errors.
- The spread of each result across tuning configurations.
Transfer probe. Nine of our emotion types have direct counterparts in GoEmotions. For those types, we test whether a model's per-emotion accuracy on generated items correlates with its per-class F1 on GoEmotions. A weak correlation would suggest that GoEmotions scores reflect recognizing emotional words more than reasoning about situations.
Decisive outcomes. The core claim holds if all three of the following occur:
- Direct-prompting accuracy falls steadily with depth, while easy-twin accuracy stays high.
- The shortcut rate rises when distractors are present.
- Isolated decomposition with model combination lowers the shortcut rate relative to the shared-context variant, at matched tokens, across the full tuning spread.
Two other outcomes would each be informative in their own way. If only rule combination helps, the main weakness lies in combining appraisals, not extracting them. If the oracle-appraisal ceiling is itself low, the models have not learned the mapping from appraisals to emotions at all.
References
- Beyond Context to Cognitive Appraisal: Emotion Reasoning as a Theory of Mind Benchmark for Large Language Models
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
- When Splitting Makes Stronger: A Theoretical and Empirical Analysis of Divide-and-Conquer Prompting in LLMs
- One Unified Model for Diverse Tasks: Emotion Cause Analysis via Self-Promote Cognitive Structure Modeling
- E-ICL: Enhancing Fine-Grained Emotion Recognition through the Lens of Prototype Theory
- A Cognitive Battery for Foundation Models: Theory-Grounded Benchmarks for Attention, Learning, Metacognition, Executive Function, and Social Cognition
- Jargon: A Suite of Language Models and Evaluation Tasks for French Specialized Domains
- GoEmotions: A Dataset of Fine-Grained Emotions