Story Morals: Surfacing value-driven narrative schemas using large language models

David G. Hobson, Haiqi Zhou, Derek Ruths, Andrew PiperEMNLP 2024 · DOI

Introduction

While Natural Language Processing (NLP) has achieved significant success in tasks like summarization and sentiment analysis, these methods often overlook the deeper "why" of a narrative. Stories serve as primary vehicles for transmitting cultural values, beliefs, and behavioral templates. However, the "morals" of these stories are rarely explicit; they are high-level abstractions derived from complex narrative events. Researchers from McGill University have introduced a new NLP task—narrative schema labeling—which involves the extraction of these "story morals." By leveraging large language models (LLMs), specifically GPT-4, they demonstrate a capacity to distill abstract, value-driven lessons from diverse texts.

Methodology: The Multi-Step Prompting Pipeline

The core challenge of moral extraction is the leap from concrete actions to abstract lessons. To address this, the researchers developed a zero-shot, multi-step prompting sequence designed to simulate "chain-of-thought" reasoning. Rather than asking for a moral in a single step, the pipeline guides the model through a structured analysis of the text.

The multi-step prompting pipeline used for moral extraction.

The process begins with summarization to focus on key plot elements, followed by the identification of protagonists and antagonists. The model then determines the central topic and the story's valence (positive or negative). Finally, it generates a free-form moral and refines it using "Positive" (Moral+) and "Negative" (Moral-) templates (e.g., "[X] is good behavior"). This templating narrows the semantic space, making the outputs more comparable and easier to analyze.

Evaluation and Datasets

The framework was validated using a dataset of 144 documents across six genres, including folktales, book summaries, and political news. To benchmark the LLM's performance, the researchers compared its outputs against human-annotated morals using both automated semantic similarity metrics (such as BERTScore) and a human "voting" paradigm. In the voting exercise, Amazon Mechanical Turk (AMT) workers were asked to select the most applicable moral from a set containing both LLM-generated and human-written options.

Additionally, the researchers applied the method to a larger dataset of 1,760 folktales from the "Fairy Talez" repository, representing 54 cultural groups. This application demonstrated the utility of moral extraction for large-scale cultural analysis.

Key Results: Human Preference and Alignment

A significant finding of the study is that human judges often preferred GPT-4’s interpretations over those written by humans. Qualitative analysis suggests that GPT-4 is more explicit about cause-and-effect relationships within the narrative. While humans might write cryptic, maxim-like statements, the LLM tends to clearly articulate how specific actions lead to specific consequences.

Human preference and agreement metrics for moral extraction.

As shown in the evaluation metrics, the GPT-generated moral was selected as the "most applicable" in 68.06% of cases. Furthermore, automated metrics indicated that GPT-4's morals were semantically closer to the "average" human interpretation than individual humans were to each other, suggesting that the model captures a broad consensus of narrative meaning.

Value-Driven Clustering and Cultural Fingerprinting

The extraction of morals allows for a unique form of document clustering. When clustering raw story text, groups typically form around surface-level features like characters (e.g., "stories about kings"). However, clustering the extracted morals reveals groups aligned with high-level virtues and vices, such as "courage," "greed," or "persistence." Using BERTopic, the researchers identified 129 distinct "value clusters," providing a granular taxonomy of narrative themes.

This method also enabled "cultural fingerprinting." By analyzing the frequency of specific morals across regions, the authors found statistically significant differences. For instance, North American folktales frequently emphasized "contentment," whereas African tales placed a higher premium on "cleverness."

Limitations and Future Directions

Despite its success, the researchers highlight several limitations. The reliance on GPT-4 introduces potential Western-centric biases, as the model is trained primarily on Western data. Furthermore, while the study included Mandarin news, the framework requires broader validation across more languages and diverse cultural annotators. Narrative interpretation is inherently pluralistic, and while the "majority view" serves as a useful benchmark, it does not represent the only valid reading of a story.

Future work may involve applying this narrative schema labeling to contemporary media, such as news or social media, to track how moral frameworks shift in response to global events like political crises or climate change.

Summary updated

BibTeX
@inproceedings{hobson2024story,
  author = {David G. Hobson and Haiqi Zhou and Derek Ruths and Andrew Piper},
  title = {{Story Morals: Surfacing value-driven narrative schemas using large language models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.723},
}
Figure