FactTrack: Time-Aware World State Tracking in Story Outlines

Zhiheng Lyu, Kevin Yang, Lingpeng Kong, Dan KleinNAACL 2025 · DOI

Abstract

Maintaining factual consistency in long-form AI-generated text remains a significant challenge for Large Language Models (LLMs). This paper introduces FactTrack, a framework that tracks "atomic facts" across a narrative timeline. By decomposing events into pre-conditions and post-conditions with specific validity intervals, FactTrack distinguishes between legitimate story progression and factual contradictions. Experimental results demonstrate that FactTrack significantly outperforms standard prompting baselines, achieving performance parity with or exceeding GPT-4 in complex narrative tracking tasks.

Introduction

Large Language Models often struggle with long-form generation, frequently "losing the plot" by introducing hallucinations or contradictions. A character might die in an early chapter only to reappear later without explanation. The root of this issue is that facts in a narrative are dynamic. Unlike static knowledge retrieval, narrative facts (e.g., "John has his laptop") are stateful—they change over time. Existing models often fail to reason over this shifting "world state." FactTrack addresses this by modeling story events as state transitions within a structured timeline.

Comparison of standard LLM reasoning vs. FactTrack's temporal state tracking.

Methodology

FactTrack utilizes a four-step pipeline to manage the world state:

  1. Decomposition: Using an LLM, complex events are broken down into directional atomic facts. These include pre-facts (conditions required before the event) and post-facts (states resulting from the event).
  2. Validity Interval Assignment: Each fact is mapped to a normalized timeline $[0, 1]$. Pre-facts are valid from the start until the event; post-facts are valid from the event onwards.
  3. Contradiction Detection: The system queries a "World State" database for overlapping facts that are logically inconsistent. This is performed using a fine-tuned Natural Language Inference (NLI) model trained on a custom dataset of over 18,000 fact pairs.
  4. World State Update: Consistent facts are added to the database. If a new fact represents a state change (e.g., a character moving locations), the validity intervals of conflicting older facts are truncated to prevent temporal overlap, ensuring the "most recent" truth prevails.

Key Results

The framework was evaluated using the WritingPrompts dataset (hierarchical story outlines) and the ContraDoc benchmark. Key findings include:

  • Superior Detection: FactTrack (using LLaMA2-7B) outperformed standard pairwise detection baselines, scoring 1.836 vs 1.452 on a 5-point contradiction scale.
  • GPT-4 Parity: FactTrack enabled smaller models like LLaMA2-7B to match raw GPT-4 performance. When applied to GPT-4, the framework significantly surpassed the base model's native capabilities.
  • Benchmark Success: On ContraDoc, FactTrack achieved an F1 score of 57.32%, vastly outperforming the GPT-4o-mini baseline of 10.48%, which suffered from extremely low recall.

Conclusion and Future Work

FactTrack demonstrates that structured temporal reasoning is essential for narrative consistency. By leveraging LLMs for decomposition and a formal algorithm for state tracking, the system overcomes the limitations of long-context window reasoning. Future work will focus on improving the decomposition bottleneck, handling multi-fact "global" contradictions, and extending the framework to non-narrative domains like technical documentation or news knowledge bases.

Summary updated

BibTeX
@inproceedings{lyu2025facttrack,
  author = {Zhiheng Lyu and Kevin Yang and Lingpeng Kong and Dan Klein},
  title = {{FactTrack: Time-Aware World State Tracking in Story Outlines}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.144},
}
Figure