TL;DR
The authors introduce MORABLES, a human-verified benchmark of 709 historical fables designed to evaluate abstract moral reasoning in Large Language Models (LLMs). While state-of-the-art models perform well on standard multiple-choice setups, adversarial testing reveals they heavily rely on superficial narrative cues and memorization rather than true moral understanding, frequently contradicting themselves when the task format changes.
What Problem Does This Paper Address?
Standard natural language understanding benchmarks (like GLUE or traditional Natural Language Inference datasets) are reaching saturation. Moreover, researchers increasingly suspect that LLMs achieve high scores on these tasks by exploiting shallow text heuristics and shortcut learning rather than demonstrating genuine comprehension.
To probe higher-order, abstract reasoning, this paper turns to moral inference using fables. Understanding the "moral of the story" requires a model to holistically integrate implicit themes, metaphorical narratives, and plot arcs, rather than simply extracting explicit text spans or matching keywords.
Methods
The core task is structured as a Multiple-Choice Question Answering (MCQA) problem where a model must select the correct moral for a given fable out of five options. To prevent models from relying on superficial shortcuts, the authors engineered four distinct types of challenging distractors:
- Similar-character morals: Authentic morals taken from other fables that feature similar characters but different plot developments.
- Trait-injected morals: Morals from different fables modified to include specific character traits from the target story.
- Feature-based morals: LLM-generated morals based strictly on character traits, completely detached from the actual plot events.
- Partial-story morals: LLM-generated morals based only on the first 10% of the fable's sentences.
To further stress-test the models, the authors introduced several dataset variants:
- True/False (TF): Reframing the task into binary evaluation.
- None of the Other Options (NOTO): Replacing the correct moral with a "None of the others" option to test for sycophancy and over-reliance on plausible-sounding answers.
- Adversarial (ADV): Modifying the fables via character swapping, trait injection, or inserting completely meaningless tautologies (e.g., prepending "A fact is a fact.") to expose positional and contextual biases.
Datasets
The paper introduces MORABLES, a curated dataset consisting of 709 short stories and fables primarily drawn from Western literary traditions (including Aesop, Phaedrus, Laurentius Abstemius, and La Fontaine).
Unlike previous narrative datasets that relied on noisy, crowd-sourced web anecdotes, MORABLES uses high-quality historical transcriptions and translations. Every single entry and generated distractor was subjected to rigorous human validation by native English-speaking graduate students to eliminate ambiguity and filter out outdated ethical values.
Key Insights and Ideas
- Severe Self-Contradiction: LLMs exhibit remarkable brittleness when the evaluation framing changes. When evaluating options individually in a True/False setup versus a multiple-choice setup, top-tier models contradict their own judgments in roughly 20% of cases.
- Scale Trumps "Reasoning": Surprisingly, reasoning-enhanced models (like GPT-o3-mini and DeepSeek R1) do not inherently solve this task better than standard models. The results suggest that model scale—rather than explicit reinforcement-learned reasoning paths—is the primary driver of performance here.
- The "First Impressions" Bias: Error analysis shows that "partial-story" morals are the most frequent trap for models. LLMs routinely over-rely on initial narrative cues rather than processing the full resolution of the plot.
- Vulnerability to Tautologies: Injecting completely meaningless phrases (like "It is what it is") significantly degrades model accuracy, revealing that models are highly sensitive to attention distractions at the very beginning or end of a text.
Key Results
The authors evaluated a wide range of open-weight and closed-source LLMs.
- Top Performers: Large closed-source models achieved the highest accuracy on the core dataset, led by Claude 3.5 Sonnet (84.8%), GPT-4o (84.0%), and Gemini 2.0 Flash (80.7%).
- Open-Weight Models: Large open models performed respectably (Llama 3.3 70B at 73.6% and DeepSeek V3 at 70.6%), while smaller open models struggled significantly (Mistral 7B scored just 28.4%).
- The NOTO Bottleneck: In the NOTO variant, models proved highly reluctant to select "None of the others," frequently opting for incorrect but plausible-sounding distractors instead.
- Automated Metrics Fail: When evaluating free-text moral generation, automated semantic metrics like BERTScore showed a very weak correlation (Pearson $r \approx 0.28 - 0.35$) with human quality ratings, highlighting the inadequacy of automated metrics for abstract text evaluation.
Limitations and Future Work
The primary limitation of MORABLES is data contamination. Because these fables are historical classics, they are widely available on the internet and were almost certainly included in the models' pre-training data. This makes it difficult to completely separate true abstract reasoning from memory retrieval.
Additionally, the current dataset is restricted to the Western literary tradition and the English language. The authors plan to expand future iterations of MORABLES to include fables from diverse global cultures to study cross-cultural moral alignment and to leverage the dataset as a framework for LLM explainability research.
Summary updated