Abstract
Evaluating the quality of AI-generated narratives has historically relied on subjective human judgment or surface-level metrics like BLEU, which fail to capture underlying narrative logic. This paper introduces Fabula Entropy Indexing (FEI), a novel framework that quantifies story coherence through the lens of information theory. By measuring the entropy of reader responses to narrative-specific questions, FEI provides an objective metric for the stability of the mental models (the "fabula") constructed by readers. Our results demonstrate that FEI effectively distinguishes between coherent narratives and those with corrupted transitional or world-building logic.
Introduction
Automated story generation remains a significant challenge in artificial intelligence. While large language models (LLMs) excel at generating grammatically correct text, they often struggle with long-term narrative consistency. Traditional automated metrics are ill-suited for creative tasks where novelty is desired over mimicry. Conversely, human evaluation is often inconsistent due to varying definitions of "coherence."
We propose that coherence is not a subjective feeling but the degree to which a text allows a reader to form a stable, non-contradictory mental model of the story world. Fabula Entropy Indexing (FEI) operationalizes this by measuring reader consensus. If a story is coherent, readers will agree on the facts of the story world; if it is incoherent, disagreement (entropy) will increase.
Methodology
FEI utilizes Shannon entropy to measure the distribution of answers to True/False questions about a narrative. The framework identifies two primary dimensions of coherence:
- Entropy of Transitional Coherence (ETC): This metric evaluates the logical progression of major plot points (kernels). It asks whether event sequences follow a causal or temporal logic.
- Entropy of World Coherence (EWC): This metric focuses on the consistency of static facts and character attributes within the story world.
The Entropy Calculation
For a given set of questions, entropy is calculated based on the proportion of readers who agree on an answer. High consensus results in low entropy, signaling a clear and coherent fabula. Conversely, a 50/50 split in answers indicates high entropy, suggesting the text was ambiguous or contradictory enough to force readers to guess.
Experimental Design
To validate FEI, we tested it against three types of datasets:
- Human-Written Stories: Classic short stories (e.g., Tolstoy, Rumi) serving as high-coherence benchmarks.
- Plotto-Generated Stories: Narratives constructed using structured narratological templates to ensure logical connectivity.
- Corrupted Baselines: Incoherent versions created by shuffling plot points (to target ETC) or swapping adjectives with antonyms (to target EWC).
Results and Discussion
The experimental results confirmed that FEI is a sensitive and reliable proxy for narrative coherence. In both transitional and world-building tests, uncorrupted stories consistently yielded lower entropy scores compared to their corrupted counterparts.
As shown in the figures, the metric is highly sensitive to even "minimalist" errors. For instance, flipping a single adjective to its antonym was sufficient to significantly spike the EWC score, indicating that readers' mental models were successfully disrupted.
Conclusion
Fabula Entropy Indexing provides a mathematically grounded path toward a rigorous science of machine storytelling. By focusing on reader consensus rather than "gold standard" references, FEI offers a way to benchmark the logical integrity of AI-generated text. Future work will focus on automating the question-generation process to reduce the need for human-in-the-loop evaluation, potentially allowing for real-time coherence monitoring during the AI generation process.
Summary updated