Beyond Fluency in Automated Narrative Evaluation

42 concepts33 papers1 proposal

Traditional natural language processing metrics are designed to measure how closely a machine-generated text matches a human reference. In tasks like translation or summarization, n-gram overlap metrics like BLEU or ROUGE serve as reasonable proxies for quality. However, as Wang and Wang argue, these metrics fail fundamentally in storytelling because a single prompt can validly result in a vast array of diverse, creative outputs. A story that is linguistically fluent can still be structurally incoherent, emotionally hollow, or simply boring.

The research direction emerging across these papers focuses on closing the "narrative gap": the distance between grammatical correctness and genuine narrative quality.

The Failure of Surface Similarity

The primary bottleneck in automated storytelling is that structural correctness does not equate to reader interest. Brenner notes that while multiagent planning can generate logically consistent plots, these systems often lack internal mechanisms to optimize for "interestingness" or dramatic quality. Similarly, Ishigaki et al. demonstrate that in specialized domains like sports commentary, traditional n-gram metrics fail to capture essential dimensions like entertainment value or contextual appropriateness.

This creates a scaling problem. As Yang et al. point out, evaluating long-form narratives (exceeding 2,000 words) currently requires slow, costly human assessment. Without reliable automated metrics, researchers cannot rapidly iterate on models designed for complex, multi-thousand-word arcs.

Measuring Coherence and Logic

If word-matching is insufficient, how do we measure if a story "makes sense"? Researchers are moving toward metrics that evaluate the underlying logical structure of a narrative, often referred to as its fabula.

Castricato et al. proposed Fabula Entropy Indexing (FEI), which measures coherence by analyzing human agreement on true/false questions about a story. The intuition is that a coherent story produces a shared understanding; if readers disagree on basic facts of the plot, the narrative is likely incoherent.

In the realm of interactive and multiplayer stories, Li et al. highlight the need to evaluate "narrative repair"—how well a system can bridge deviations caused by players without feeling like disruptive filler. This suggests that quality is not just about the final text, but the seamlessness of the underlying causal logic.

Quantifying Engagement and Emotion

Beyond logic, a story must be compelling. This requires measuring abstract qualities like suspense, surprise, and emotional resonance—features that n-grams cannot see.

  • Suspense as Uncertainty: Wilmot and Keller treat suspense as "uncertainty reduction." By measuring how much each new sentence narrows down the possible future directions of a story, their model can predict human-perceived suspense and identify major turning points.
  • Aesthetic Value: Ontanon and Zhu emphasize that evaluation frameworks must differentiate between narratives that are merely valid and those that are creative, shifting the focus toward the aesthetic and engagement value of the text.
  • Emotional Alignment: For personalized content, Lin et al. introduced the FICL-Score, an LLM-based metric that prioritizes emotional resonance and linguistic style over literal word-matching.

The Shift Toward Model-Based Evaluation

As Large Language Models (LLMs) become more sophisticated, researchers are increasingly using them as evaluators to bridge the gap between automated metrics and human judgment. Chen and Si developed an "Annotator-Critic" loop where one agent extracts narrative details and another attempts to reconstruct the story, using the success of that reconstruction as a measure of narrative clarity.

However, even these advanced methods face challenges. Teleki et al. observe a persistent divide between automated linguistic metrics and small-scale user studies that lack statistical generalizability. Whether evaluating theatre scripts (Rosa et al.), financial reports (Li et al.), or visual stories (Lotfi et al.), the field is converging on a single requirement: metrics must move past the surface of the text to understand the psychological and artistic dimensions of the narrative.

Key Foundational Work

Creating Dynamic Story Plots with Continual Multiagent Planning (2010) — contributed 2 concept elements to this direction.
On the Role of Domain Knowledge in Analogy-Based Story Generation. (2011) — contributed 1 concept elements to this direction.
Robust and Authorable Multiplayer Storytelling Experiences (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_narrative_quality_assessment_4819c90b,
  title = {narrative quality assessment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/narrative-quality-assessment/rdr_4819c90b41d54fc18af7014750ba0e88},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_narrative_quality_assessment_4819c90b_v1,
  title = {narrative quality assessment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/narrative-quality-assessment/rdr_4819c90b41d54fc18af7014750ba0e88/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake