Fabula Entropy Indexing: Objective Measures of Story Coherence

Louis Castricato, Spencer Frazier, Jonathan Balloch, Mark Riedl

TL;DR

This paper introduces Fabula Entropy Indexing (FEI), a novel method for objectively measuring story coherence by analyzing human agreement on true/false questions about a story. FEI offers a theoretically grounded approach to overcome the limitations of traditional automated metrics, providing a more reliable way to evaluate story generation systems.

What Problem Does This Paper Address?

Automated story generation is a challenging area in AI, primarily due to the lack of robust, objective measures for evaluating the coherence of generated stories. While traditional metrics like perplexity and BLEU assess linguistic fluency or similarity to existing text, they fail to capture the nuanced quality of a compelling narrative. Coherence, in particular, is difficult to quantify because it can be interpreted differently by individuals. This ambiguity makes it hard to definitively say whether an AI-generated story is truly "good" or just grammatically sound. The paper aims to bridge this gap by providing a method that leverages human judgment in a structured, quantifiable way to assess story coherence.

Methods

The core of Fabula Entropy Indexing (FEI) lies in its reliance on human participants answering true/false questions about a story. The key insight is that coherent stories lead to more consistent mental models among readers, resulting in higher agreement on answers. Conversely, incoherent stories force readers to guess, leading to more disagreement and thus higher "entropy" in their responses.

FEI decomposes coherence into two main metrics:

  • Entropy of Transitional Coherence (ETC): This metric focuses on the sequential coherence of events. It assesses whether a reader could reasonably anticipate an event occurring given a preceding one, essentially measuring the necessity of transitional ordering. ETC questions are framed around "kernels" (major plot points) and probe the relationship between events before and after these kernels.
  • Entropy of World Coherence (EWC): This metric measures the internal consistency of the story's world. It evaluates the coherence of static facts about characters, objects, and locations, regardless of temporal order. EWC questions focus on descriptive elements (adjectives, adverbs) and their consistency within the narrative.

The methodology involves:

  1. Generating stories (either human-written or AI-generated).
  2. Creating a set of true/false questions designed to probe transitional or world coherence. These questions are not about factual correctness but about the reader's inferred understanding of the story's internal logic.
  3. Recruiting human participants to answer these questions for each story.
  4. Calculating the entropy of the answers (i.e., the disagreement among participants) for each question and aggregating it to get an overall coherence score for the story. Lower entropy indicates higher coherence.

Datasets

The paper uses two main sources for evaluation:

  • Human-written short stories: Sourced from a public repository, these are assumed to be coherent and serve as a baseline.
  • Corrupted stories: These are created by intentionally altering human-written stories to introduce incoherence. For EWC, this involved swapping adjectives and adverbs for their antonyms. For ETC, stories generated using the Plotto technique (which provides plot point sequences) were corrupted by shuffling the order of plot points.

The paper does not introduce a new benchmark dataset but rather uses existing stories and a novel corruption methodology to validate its proposed metrics.

Key Insights and Ideas

  • Human Agreement as a Proxy for Coherence: The fundamental idea that reader agreement on simple questions directly reflects story coherence is a powerful insight. It shifts the focus from objective truth to subjective consistency, which is crucial for narrative understanding.
  • Decomposition of Coherence: Separating coherence into "transitional" and "world" aspects provides a more nuanced understanding of narrative quality. This allows for targeted evaluation of different types of narrative flaws.
  • Entropy as a Measure of Disagreement: Using entropy to quantify the level of disagreement among readers is a mathematically sound and intuitive way to capture the "randomness" introduced by incoherence.
  • FEI's Independence from Ground Truth: A significant advantage is that FEI does not require a pre-defined "correct" answer for questions. This makes it applicable to a wide range of stories, including fictional ones where objective truth is irrelevant.

Key Results

The experiments demonstrated that FEI successfully distinguishes between coherent and incoherent stories.

  • Uncorrupted stories consistently showed lower entropy values for both ETC and EWC compared to their corrupted counterparts. This indicates that the proposed metrics are sensitive to intentional incoherence.
  • Corrupted stories exhibited higher entropy, meaning participants disagreed more on the answers, suggesting a breakdown in their mental models of the story.
  • The paper also notes that while human-written stories generally have lower entropy than AI-generated stories (as observed in a separate, unpublished evaluation), the corrupted stories showed a clear increase in entropy, validating the method.

Limitations and Future Work

  • Question Generation: The quality of the entropy indices is highly dependent on the quality and relevance of the true/false questions. While the paper used human question writers, it acknowledges that the authors themselves might have generated better questions, highlighting the potential for experimenter bias. Automating the generation of effective questions remains an open research problem.
  • Story Length and Complexity: FEI was validated on short stories (10 sentences or less). Applying it to longer narratives would require substantially more questions and participant effort.
  • Descriptive Nature of Stories for EWC: EWC relies on descriptive elements (adjectives, adverbs) that are integral to the narrative. Story generators that produce less descriptive text might not be as effectively evaluated by EWC.
  • ETC vs. EWC: The paper suggests that ETC might be a less robust metric than EWC, as uncorrupted stories showed higher ETC values, potentially due to human tolerance for event ordering variations or limitations in the Plotto corruption method.

Future work could focus on automating question generation, exploring FEI's applicability to longer and more complex narratives, and refining the ETC metric. The authors also suggest that FEI has been successfully used to evaluate an automated story generation system, demonstrating its practical utility.

Summary updated

BibTeX
@inproceedings{castricato2021fabula,
  author = {Louis Castricato and Spencer Frazier and Jonathan Balloch and Mark Riedl},
  title = {{Fabula Entropy Indexing: Objective Measures of Story Coherence}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure