Introduction
Standard neural language models often struggle with long-form narrative coherence. While they excel at local fluency—making individual sentences sound natural—they frequently lose track of characters, swap genders mid-paragraph, or fail to maintain the state of objects. This paper introduces ENGEN (Entity-based Generation), a model designed to maintain dynamic "entity representations" to ensure stories remain consistent as they unfold.
The Challenge of Narrative Coherence
The authors argue that the primary missing ingredient in current models is explicit entity tracking. In a story, characters (entities) serve as the anchors of coherence. If a model fails to recognize that "Emily" is the same person referred to as "she" later in the text, or that her state has changed after an interaction, the narrative becomes disjointed.
Methods: Integrating Entity Context
ENGEN builds upon a sequence-to-sequence architecture by integrating three distinct sources of context during word generation:
- Local Context: The standard RNN hidden state representing the current sentence.
- Recent Context: An attention mechanism that looks back at the previous sentence to maintain immediate flow.
- Entity Context: The core innovation. Every entity mentioned is assigned a dynamic vector representation that updates whenever they are mentioned, capturing their evolving state in the story.
To merge these inputs, the researchers utilized a max-pooling function, which effectively selects the most salient features from local, recent, and entity-specific contexts at each step.
Datasets and Evaluation
The model was trained on the Toronto Book Corpus, comprising 312 adventure books (approximately 43 million tokens). Stanford CoreNLP was used to automatically annotate entity mentions and coreference chains.
A key contribution of this work is the "Mention Generation" task. Instead of simply predicting the next word, the model must choose the correct entity to mention and the appropriate linguistic form (e.g., deciding between "Emily," "the girl," or "she").
Key Results
The inclusion of entity context significantly improved performance across multiple metrics:
- Mention Generation: ENGEN achieved a Mean Average Precision (MAP) of 0.53, a substantial improvement over the entity-unaware S2SA baseline, which scored 0.44.
- Sentence Selection: In a "gold vs. distractor" test, ENGEN correctly identified the real next sentence with 56.6% accuracy, outperforming baselines with statistical significance.
- Human Evaluation: In a blind comparison, human evaluators preferred ENGEN-generated sentences in 27 out of 50 cases. Qualitative feedback indicated that humans particularly valued the model's ability to correctly link pronouns to characters mentioned much earlier in the narrative.
Insights and Limitations
The study found that while entity tracking is vital, it must be balanced with immediate context; models that ignored the preceding sentence performed worse overall. Furthermore, the use of max-pooling proved surprisingly effective for merging disparate data types compared to more complex methods like averaging or multiplication.
However, the model remains limited by the quality of automatic coreference resolution. Noisy annotations during training can lead to incorrect entity associations. Additionally, while ENGEN improves structural coherence, it still occasionally misses "social context"—generating sentences that are grammatically correct but out of character for the established tone. Future work may involve more complex story planning to help models understand the underlying motivations behind character actions.
Summary updated