TL;DR
This paper introduces MORABLES, a new benchmark designed to evaluate the abstract moral reasoning capabilities of Large Language Models (LLMs) using fables. The findings reveal that while larger LLMs perform better, they still struggle with genuine moral inference, often relying on superficial patterns and contradicting themselves when faced with adversarial challenges.
What Problem Does This Paper Address?
As LLMs become increasingly proficient in standard reading comprehension tasks, the focus is shifting towards evaluating their ability to perform complex, abstract reasoning. Traditional benchmarks, like those for Natural Language Inference (NLI), have shown limitations, with models often exploiting shallow heuristics rather than demonstrating true understanding. This paper addresses the need for a benchmark that probes deeper comprehension skills, specifically abstract moral reasoning, using rich narrative content. Fables, with their inherent moral depth and concise storytelling, are ideal for this purpose, offering a more authentic and challenging evaluation than many existing datasets.
Methods
The core of the MORABLES benchmark is a Multiple Choice Question Answering (MCQA) task. Each fable is paired with a human-verified moral and five answer choices, including carefully crafted distractors designed to challenge LLMs beyond simple keyword matching. To further stress-test model robustness, the authors introduce adversarial variants of the fables and multiple-choice options. These variants include:
- Character Swap: Replacing characters with similar ones.
- Trait Injection: Inserting key traits of characters into the moral.
- Feature-Based Moral: Generating a moral based solely on character traits, detached from the story's context.
- Partial-Story Moral: Generating a moral based on an excerpt of the story.
- Tautology Injection: Adding meaningless tautological sentences to the story.
Additionally, the paper explores variants like "None of the other options" (NOTO) and True/False (TF) framing, and includes a free-text moral generation evaluation.
Datasets
The primary dataset, MORABLES, consists of 709 fables and their corresponding morals, sourced from historical literature, with a strong emphasis on Aesop's fables. The dataset was manually verified and curated.
The paper also introduces several variants for evaluation:
- Core MORABLES: The original MCQA dataset.
- NOTO Variant: Correct answers are replaced with "None of the other options."
- TF Variant: The task is reframed as a True/False question.
- ADV Variant: Incorporates adversarial modifications to fables and choices.
Key Insights and Ideas
- Scale Matters, But Isn't Everything: Larger LLMs significantly outperform smaller ones, but even the most advanced models exhibit brittle reasoning and can be easily fooled by adversarial examples.
- Superficial Cues are a Weakness: Models frequently rely on shallow patterns, such as character names or story excerpts, rather than genuine moral understanding. This is evident in their susceptibility to "partial-story" and "feature-based" distractors.
- Self-Contradiction is Common: The best models can contradict their own judgments in a significant percentage of cases (around 20%) when the task framing is slightly altered, highlighting a lack of robust reasoning.
- Reasoning-Augmented Models Don't Always Win: Explicit reasoning capabilities do not necessarily translate to better performance; model scale appears to be a more dominant factor.
- Human vs. Automated Evaluation: Semantic similarity metrics like BERTScore show a weak correlation with human judgment of moral quality, underscoring the subjective and nuanced nature of moral assessment.
Key Results
- Performance Gap: The best large open models (Llama 3.3 70B, DeepSeek V3) achieved accuracies around 70-73%, while smaller models struggled with accuracies below 47%. Closed-source models like GPT-4o and Claude 3.5 Sonnet led with accuracies around 80-84%.
- Adversarial Impact: Adversarial modifications, particularly tautology injections, significantly decreased model performance, with accuracy dropping by up to 12.3% for GPT-40.
- NOTO Variant Findings: Models showed a strong reluctance to select "None of the others," often choosing a plausible but incorrect moral, suggesting a sycophancy bias or a preference for internal probability rankings over strict correctness.
- Free-Text Moral Generation: While LLMs generated morals that generally aligned with fables, human ratings showed considerable variability, and BERTScore F1 values were low, indicating a divergence in wording from original references.
Limitations and Future Work
- Memorization Concern: The primary limitation is the high likelihood that LLMs have encountered these fables and morals during pre-training, raising concerns about memorization rather than genuine inference. Future work could involve creating entirely new stories.
- Cultural Bias: The dataset is predominantly sourced from Western traditions, limiting the study of cultural biases in moral reasoning. Future work aims to expand the dataset to include fables from diverse cultures.
- Model Evolution: The rapid release of new models makes it challenging to keep benchmarks up-to-date.
- Explainability: The paper suggests future work on leveraging the holistic nature of moral lessons for explainability research.
Summary updated