Modeling Fair Play in Detective Stories with Language Models

Eitan Wagner, Renana Keydar, Omri AbendarXiv 2025 · DOI · arXiv

Abstract

This paper introduces a formal probabilistic framework for "Fair Play" in detective fiction—the unwritten contract where an author must provide enough clues for a reader to solve the mystery while ensuring the resolution remains surprising. By defining various reader models and applying information-theoretic metrics, the researchers evaluate the structural logic of mysteries generated by Large Language Models (LLMs) compared to classic literature. The findings suggest that while LLMs can mimic the prose of a "whodunit," they struggle to balance coherence and surprise, often resulting in illogical "Deus ex Machina" endings.

Introduction: The Fair Play Dilemma

In detective fiction, the author navigates a delicate tension. If a story is too coherent, the reader identifies the culprit immediately, resulting in low surprise. Conversely, if the ending is entirely unexpected but lacks supporting clues, the story feels "unfair" or illogical. This balance is known as Fair Play.

While literary quality is often viewed as subjective, this research bridges the gap between narratology and computational linguistics. The authors ask: Can "Fair Play" be mathematically defined? And can current LLMs master the structural logic required to play fair with a reader?

Formalizing the Reader

To quantify these narrative dynamics, the researchers propose a framework involving three distinct "Reader Models" that simulate different levels of intelligence and access to information.

Hierarchy of reader models categorized by intelligence and perspective.
  1. The Gullible Reader ($M_0$): A naive internal model easily misled by "red herrings" or distractors.
  2. The Brilliant Detective ($M_1$): An internal model performing optimal Bayesian inference based strictly on the clues provided in the text.
  3. The Know-it-all Reader ($M_\infty$): An external model with access to the "story generation distribution," essentially knowing the author's stylistic patterns and tropes.

Metrics of Mystery

The framework utilizes several key metrics to evaluate story quality:

  • Informedness: Uses cross-entropy to measure how much a reader knows about the culprit at any point in the narrative.
  • Fair Play Score ($S_{FP}$): The difference between the coherence score (what the Know-it-all knows) and the surprise score (how much the Gullible reader is fooled). A high score indicates a story that is surprising yet logical in hindsight.
  • Expected Revelation Content (ERC): A metric testing clue effectiveness by masking text and asking a model to predict it based on the final revelation.

The Dynamics of Surprise and Coherence

A central finding of the paper is a mathematical theorem stating that a single reader model cannot be both "strongly surprised" and "intelligent" simultaneously. For a mystery to succeed, the reader must undergo a cognitive shift.

Probability of identifying the culprit vs. distractor for Gullible and Know-it-all readers throughout the story stages.

As shown in the figure above, a well-constructed mystery keeps the probability of the true culprit low for the gullible reader during the "Suspicion" phase, often favoring a distractor. Upon the "Revelation," the probability for the true culprit must spike to near-certainty, validating the clues provided earlier.

Evaluating AI-Generated Mysteries

The study compared classic works by Arthur Conan Doyle and Agatha Christie against stories generated by models like Llama-3, Gemini 1.5, and GPT-4o.

  • LLM Failure in Fair Play: Most LLMs failed to achieve a positive Fair Play score. Smaller models often created "Deus ex Machina" endings where the culprit was revealed without any logical connection to previous clues.
  • The "Jameson" Effect: LLMs often suffer from "mode collapse," where stylistic tics act as spoilers. For instance, Llama models frequently named the detective "Jameson," allowing a "Know-it-all" reader to solve the mystery through pattern matching rather than clue analysis.
  • Scaling and Validity: While larger models like GPT-4o showed better "Generation Validity" (following instructions to include suspects and distractors), they still struggled to plant "effective" clues that met high ERC standards.

Conclusion and Future Directions

The research highlights a significant limitation in current AI: the inability to maintain deep, long-range structural logic in complex narratives. While LLMs excel at surface-level prose, the "Fair Play" required for a satisfying mystery remains elusive. Future work aims to validate these information-theoretic metrics against actual human reader predictions to further refine the "Gullible Reader" model and its capture of human suspense.

Summary updated

Abstract

Good storytelling involves surprise -- unpredictability in how the story unfolds -- and sense-making, the requirement that the story forms a coherent sequence. However, to date, these two qualities have largely been addressed in isolation. We formalize these qualities and their relationship in an information-theoretic framework, using detective fiction as a paradigm case of narratives in which a hidden truth is discovered through reasoning. Our central theoretical result shows that surprise and coherence must trade off for any *single* reader model, but can coexist when two reader modes are distinguished: a pre-revelation mode that forms expectations while the ending is unknown, and a post-resolution hindsight mode that re-evaluates the story after the culprit is revealed. The balance of these two dimensions is realized in the common requirement of *fair play*, giving the reader a chance to solve the mystery while maintaining a challenge. We operationalize the framework using large language models as simulated readers, and define reference-less evaluation metrics for surprise, coherence, and fair play. Experiments on LLM-generated stories validate our theoretical predictions: while models generally succeed in creating surprise or coherence, achieving fair play poses a challenge even for strong models. Moreover, surprise and coherence do not positively correlate across stories, resisting reduction to a single latent quality. A human study validates the metrics, confirming they capture aspects of narrative quality that matter to readers. Our metrics also reproduce established literary intuitions, finding Christie's stories more surprising and more fair-playing than Conan Doyle's.
BibTeX
@misc{wagner2025modeling,
  author = {Eitan Wagner and Renana Keydar and Omri Abend},
  title = {{Modeling Fair Play in Detective Stories with Language Models}},
  year = {2025},
  eprint = {2507.13841},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.13841},
}
Figure