Language models can write fluent paragraphs and still lose track of who did what to whom over a long story. Measuring that failure honestly is hard for two reasons. First, the long-story benchmarks we have are built from published novels and public documents, which models may already have memorized. Second, those benchmarks are almost all in English or Chinese, languages where plot-relevant information such as "because", "after", or "the dog bit the man" sits in separate words.
In agglutinative languages like Turkish or Korean, which build words by chaining many suffixes onto a root, the same information often sits inside a single word: a case suffix marks who is the agent, a verb suffix marks that one event caused another. This proposal builds a generator that produces fresh, never-before-seen stories from a hidden graph of events and character states. It then writes each story out in English, Turkish, and Korean, with direct control over whether each decisive clue appears as a free word or as a bound suffix. The central question is whether long-range story tracking breaks down specifically when the clue is carried by morphology, and whether static benchmarks have been hiding this failure because memorized plots let models skip the tracking altogether.
Core Claim
We claim that one variable, which we call morphological load, governs a measurable part of long-context narrative failure. Morphological load is whether the single piece of text that a correct answer depends on is a free word or a bound morpheme (a suffix or other affix that cannot stand alone). We predict that this effect has been invisible for two reasons. Existing benchmarks are English-centric. They are also static, so contaminated models can reach answers through remembered tropes rather than through tracking.
Two things would be new. The first is a dynamic, graph-grounded story generator, which extends the logic of dynamic evaluation from formal reasoning into narrative. The second is a controlled manipulation of where coherence clues live inside words, which no existing benchmark allows.
Positioning Against Prior Work
The work that motivates this proposal found gaps without closing them. DetectiveQA tests reasoning over full detective novels. It shows a gap between answer accuracy and reasoning quality: Claude 3 Opus reaches 81.95% multiple-choice accuracy but only a 37.33% step-wise reasoning score. The authors attribute part of this gap to shallow heuristics and memorized tropes. Because its novels are published and static, it cannot separate tracking from recall, and it covers only English and Chinese.
HELMET shows that synthetic needle-retrieval scores do not predict real long-context ability. It relies on static English sets such as NarrativeQA (Kociský et al.) and leaves multilingual evaluation to future work. MGAL does go multilingual. It finds that models abandon discourse roles in favor of local lexical cues. However, it uses bureaucratic UN reports, which are public and therefore contaminable, and it includes no agglutinative language.
The method we borrow comes from DyVal. DyVal generates reasoning problems on the fly from directed acyclic graphs, meaning networks of steps where each step depends only on earlier ones, and it varies graph depth and presentation order. It shows that models which look strong on static benchmarks collapse on fresh instances; for example, GPT-3.5-Turbo loses 23% arithmetic accuracy from its easiest to its hardest tier. We keep DyVal's recipe of a hidden graph plus controlled verbalization. We replace arithmetic nodes with events, character states, and causal and temporal links, and we add language and morphological placement as verbalization controls. DyVal explicitly stops at formal tasks.
The work that constrains our design is Data Contamination Can Cross Language Barriers. It shows that fine-tuning on translated benchmarks inflates English scores; in one case MathQA for LLaMA3-8B trained on French rose from 42.01% to over 95%, with n-gram and perplexity detectors registering nothing. It also finds weaker transfer from Korean than from European languages and attributes this loosely to subword vocabularies. Two consequences follow for us. Translating a static English benchmark into Turkish does not make it clean, so freshness must come from generation. The same paper also supplies the injection protocol we reuse to check that our benchmark actually detects contamination.
The morphology literature supplies the linguistic units we manipulate. In the Turkish treebanks, the dependency relations that matter are carried by inflectional groups, the chunks of a word between derivational boundaries (Sulubacak & Eryiğit). Universal Dependencies instead treats most suffixes as feature bundles attached to a content word (Universal Dependencies for Turkish). Korean analysis faces surface forms where distinct morphemes merge into identical strings (Rich Character-Level Information for Korean Morphological Analysis). We use these analyses to decide which clues can be moved between a free-word position and a bound-morpheme position, and to verify our generated text.
Research Program
The story world. The generator samples a story world, meaning a set of characters with attributes and a sequence of events. Each event records an agent, a patient (the one acted upon), a location, a time, and optional causal or temporal links to earlier events. Every question is a query against this graph. Examples are "Who held the key when the fire started?" and "Did the letter arrive before or after the argument?" Each question has a gold reasoning chain, which is the path of graph nodes needed to answer it.
Because the chain is known, we can score reasoning without an LLM judge. We check whether the model's stated intermediate facts match nodes on the path. This removes the cost and position bias of judge-based scoring that HELMET notes.
We control difficulty through three parameters. Depth is the number of events in the chain. Distance is the number of tokens between clue and question. Distractor density follows MGAL's "local semantic crowding," so we place unrelated events about the same characters near the clue.
Verbalization. A verbalizer turns each graph into prose from a template grammar. For Turkish, it runs a finite-state morphological generator plus lexicon; for Korean, it uses a morphological generator; for English, templates alone. Filler narration from a separate grammar pads stories out to 8K, 32K, and 128K tokens.
For each clue type, the verbalizer can emit a free-word form or a bound form. The clue types are agent–patient role, causation, temporal order, and evidentiality or hearsay. Here are the pairs. For role, Turkish can mark who acted with word order plus an explicit postpositional phrase, or purely with accusative and dative case suffixes under scrambled order. For causation, it can use a free connective such as "çünkü" ("because"), or a converb suffix (a suffix that turns a verb into an adverbial clause) such as "-ince" or "-diği için". For evidentiality, it can use a free lexical phrase meaning "reportedly", or the "-miş" suffix. Korean has analogous pairs through its particles and connective endings. English acts as the all-free-word reference.
Holding the graph fixed and flipping only placement gives minimal pairs. These isolate morphological load from content, length, and difficulty.
Quality control. Every Turkish and Korean story is parsed. We parse Turkish with the spaCy transformer pipeline from Altinok's Turkish resources, which reports a dependency parsing unlabeled attachment score of 0.79, and Korean with a character-level morphological analyzer. We check that each clue morpheme is recoverable and that no unintended clue leaks through. Native speakers rate a stratified sample for grammaticality and naturalness. Items whose paired versions are rated as unequally natural are dropped, because a stilted suffix construction would confound "morphology is hard" with "the text is weird".
Trope-conflict variants. To target the recall shortcut directly, we build a third family of items. Each graph is seeded from a common narrative trope, for example "the butler did it". Its counterfactual twin changes one edge so that the trope-consistent answer becomes wrong. This is a narrative analogue of Choice Confusion. A model that tracks state should be indifferent to the swap, while a model leaning on remembered story patterns should fail on the twin.
Baselines
Question-only reference. The naive reference point is the same model answering each question with the story removed. This measures how much can be guessed from the question and priors. MGAL reports this ablation on its own data, with accuracy falling from 0.73 with context to 0.31 without. We expect question-only accuracy on our trope-conflict twins to sit below chance, since priors actively mislead there. Any condition that does not clearly beat question-only accuracy counts as a failure of tracking, not a success.
Static narrative benchmarks. The strongest existing evaluation competitor is scoring the same models on DetectiveQA and on HELMET's NarrativeQA subset, in their original languages and machine-translated into Turkish and Korean. The reported anchor is Claude 3 Opus on DetectiveQA, with 81.95% multiple-choice accuracy and a 37.33% reasoning score. We will run open models of the same families with identical prompts and context lengths on both the static sets and our generated sets.
Our benchmark "beats" static evaluation if two things hold. First, it ranks models differently from the static sets. Second, and more importantly, its scores do not move when a model is deliberately contaminated, while the static-set scores do. No reported result was found on Lacuna for any model on long-context narrative reasoning in Turkish or Korean.
Distilled-state prompting. The strongest existing method for keeping stories coherent is the recursive-reprompting idea from Re3. There, the model works from a running summary and structured character attributes rather than raw history. Re3 reports a 14% absolute gain in human-judged plot coherence on WritingPrompts. We adapt it to comprehension: the model reads the story in chunks, updates a character-attribute state after each chunk, and answers from that state. Chunking and token budgets will match the raw long-context run.
This baseline matters because it may absorb the morphological penalty. If the per-chunk state extraction handles suffixes well locally, the long-range cost of bound clues could vanish. That is a legitimate and informative outcome.
Key-ingredient ablation. The ablation that isolates our main variable is the free-versus-bound minimal pair, holding graph, length, and distractors fixed. A related intervention runs the bound condition with the input pre-segmented at morpheme boundaries by the analyzer. It tests whether any penalty comes from the language itself or from how subword tokenizers fragment the words.
Why not just…?
Why not just translate DetectiveQA or NarrativeQA into Turkish? Translation does not remove contamination: Data Contamination Can Cross Language Barriers shows memorized content transfers across languages without triggering surface detectors. A translated static benchmark also offers no minimal pairs, so any Turkish drop could come from translation quality, length inflation, or morphology in unknown proportions. We include the translated sets as a baseline precisely to show this confound empirically. They should respond to injected contamination, while our generated items should not.
Why not just verbalize DyVal's arithmetic and logic graphs in Turkish? This is a real competitor for the morphology question, and we run it as a control. If the bound-clue penalty appears equally on DyVal-Turkish, then the effect is a generic morphological reading cost, and the narrative machinery is unnecessary for that finding. Our prediction is that the penalty is larger for narrative items for two reasons. Roles and causation must be carried across thousands of tokens of distractor-dense text. And DyVal found that depth, not width, drives failure, which our distance manipulation stresses. If the prediction fails, we report the simpler result.
Why not just have an LLM write fresh stories and have a judge score them? LLM-written stories are fresh as strings but not as plots; they reproduce common tropes, which is exactly the shortcut we want to remove. They also have no gold state graph, so scoring falls back on expensive, biased judges. The graph-grounded design gives exact reasoning-chain scoring and controllable placement. The cost is naturalness, which we handle through native-speaker filtering and by comparing results on a small set of human-written stories annotated with graphs.
Why not just fix tokenization and assume morphology stops mattering at scale? This is a hypothesis we test, not one we assume. The segmentation intervention and the scale sweep, run on the same minimal pairs, would show whether the penalty shrinks with model size or disappears under morpheme-aligned input. If it vanishes at the largest scale with standard tokenizers, that is a clean negative result.
Proposed Experiments
We evaluate open model families at several sizes with native long context, plus API models where licensing allows, over depth 2–8, distance 8K–128K, three languages, and the two placement conditions.
Test 1: interaction. The first decisive test is whether the accuracy gap between bound and free clues in Turkish and Korean grows with clue-to-question distance and depth, while English stays flat at matched difficulty. A constant gap would mean a local reading cost. A growing gap would mean suffix-carried plot state fades over long spans, which is the novel claim.
Test 2: validating contamination-proofness. Following the injection protocol of Yao et al., we fine-tune a model on Turkish and on French renderings of a fixed pool of generated stories and questions. We then test on three sets: English renderings of the same graphs, fresh graphs, and trope-conflict twins. Contamination should inflate the same-graph scores but not the fresh-graph scores, and the twins should expose it. We predict that injection through Turkish inflates English scores less than injection through French, consistent with the Korean result in that paper. We then check whether the weaker transfer concentrates on items whose clues were bound morphemes, which would give a morphological account of the typological asymmetry that paper left unexplained.
Test 3: masking. The third test asks whether model rankings on static narrative benchmarks predict rankings on fresh bound-clue items. It also asks whether the trope-conflict drop correlates with each model's static-benchmark advantage.
Test 4: mechanism. Finally, we compare raw long context, distilled-state prompting, and morpheme-segmented input on the bound condition. This tells practitioners whether planning-and-state methods or tokenization changes are the lever for agglutinative long-form text.
All metrics are exact-match answer accuracy and graph-grounded chain precision and recall. We report confidence intervals from resampling graphs, not items, so that correlated questions about the same story do not overstate certainty.
References
- DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels
- HELMET: How to Evaluate Long-context Models Effectively and Thoroughly
- The NarrativeQA Reading Comprehension Challenge
- MGAL: A Multilingual Granularity-Aware Long-Context Benchmark
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Data Contamination Can Cross Language Barriers
- Representation of Morphosyntactic Units and Coordination Structures in the Turkish Dependency Treebank
- Universal Dependencies for Turkish
- Rich Character-Level Information for Korean Morphological Analysis and Part-of-Speech Tagging
- A Diverse Set of Freely Available Linguistic Resources for Turkish
- Re3: Generating Longer Stories With Recursive Reprompting and Revision