TL;DR
This paper introduces a novel character-centric framework for analyzing moral foundations in narratives using Large Language Models (LLMs). By developing a specialized questionnaire (MFCAQ) and validating it against human annotations, the authors demonstrate LLMs' potential for scalable moral reasoning in stories, revealing nuanced cross-cultural patterns in moral expression.
What Problem Does This Paper Address?
Understanding how moral values are conveyed through stories is crucial, as narratives are primary vehicles for disseminating cultural norms and ethical frameworks. While Moral Foundations Theory (MFT) offers a robust framework for categorizing moral reasoning, its application to computational narrative analysis has been limited. Existing computational approaches often rely on lexicon-based methods that struggle with context and ambiguity, or focus on abstract moral themes rather than the concrete actions of characters. This paper aims to bridge this gap by developing a character-focused method to quantify moral foundations in narratives, enabling large-scale analysis of how these values are embedded and expressed across diverse cultures.
Methods
The core of this work is the development of a novel Moral Foundations Character Actions Questionnaire (MFCAQ). This questionnaire operationalizes the five original MFT foundations (Authority, Care, Fairness, Loyalty, Purity) by creating 16 items designed to assess character actions and motivations within a narrative context. Crucially, the MFCAQ captures both positive and negative valences for each foundation, allowing for a more nuanced understanding of moral expression than a simple binary approach.
The authors then leverage Large Language Models (LLMs) to automate the application of the MFCAQ. They experimented with various LLMs, including GPT-4 and several open-source models, and found that explicitly referencing MFT in prompts improved performance. To validate their approach, they compared LLM annotations against human annotations on a subset of stories, using Spearman's rank correlation. They also developed a character valence scale to assess portrayals from "villain" to "hero."
Datasets
The primary dataset used in this study is the Kaggle Folk Tales dataset, comprising 2,697 traditional stories from 55 countries. This dataset was chosen for its rich representation of cultural values, norms, and didactic structures. The authors acknowledge that the dataset's reliance on English translations is a limitation.
Key Insights and Ideas
- Character-centric analysis is key: Focusing on character actions, rather than abstract themes, provides a more direct and scalable way to analyze moral content in narratives.
- LLMs are capable of nuanced moral reasoning: The study demonstrates that LLMs, when guided by a well-designed questionnaire like the MFCAQ, can perform complex moral annotation tasks with reasonable accuracy, correlating well with human judgments.
- Moral foundations are universally present but culturally nuanced: The findings suggest that moral foundations are widely distributed across cultures and stories, but with significant regional variations in emphasis.
- Folktales offer a balanced moral landscape: Contrary to previous claims of a strong positivity bias, the study reveals a more balanced distribution of positive and negative moral content in folktales, challenging the notion that traditional narratives solely promote positive instruction.
Key Results
- LLM Performance: The study found moderate to strong correlations between LLM predictions and human annotations across all MFT dimensions. GPT-4 showed the strongest overall performance, with competitive results from Gemma2 despite its smaller size.
- Over-prediction Bias: LLMs exhibited a systematic bias towards over-identifying the presence of moral content, assigning higher relevance scores than human raters. This bias was addressed using a simple bias-adjustment method.
- Content Relevance: The MFCAQ proved effective, with 98.63% of stories including at least one moral foundation and 76.83% exhibiting three or more.
- Positivity Bias: While folktales generally skew positive in character portrayal (54.6% rated 4 or 5 on a 5-point scale), there was no uniform positivity bias across all moral foundations.
- Regional Variations: Bootstrapping analysis revealed meaningful regional variations in moral foundation emphasis, with approximately one-third of region-foundation pairs showing statistically significant deviations from global means. PCA analysis highlighted polarity as a key differentiator of regional behavior.
Limitations and Future Work
The authors acknowledge several limitations:
- Dataset Scope: The study exclusively uses folktales, which are often shorter and more morally explicit than longer, modern narratives. This raises questions about the generalizability of findings to more complex texts.
- Translation: The reliance on English translations for stories originally composed in various languages may have altered moral emphases.
- Geographical and Temporal Scope: While the dataset covers 55 countries, regional biases in story selection and translation limit its representativeness.
- Granularity of Foundations: The condensed nature of folktales might limit the granularity needed to distinguish between specific moral foundations, potentially leading to correlations driven by general polarity.
Future work suggested includes:
- Applying the framework to more complex narrative forms (novels, films).
- Developing methods to account for character diversity in longer narratives.
- Exploring multilingual models to address translation issues.
- Investigating the effect of model scaling and architecture on performance.
- Exploring alternative phrasing and prompting for character attributes, and considering latent moral dimensions beyond MFT.
- Diversifying the annotator pool and manually coded stories for richer cultural calibration.
Summary updated