Judging Long Stories by Retrieving What Each Step Assumes

When a language or vision-language model writes a long story, a multi-step recipe, or an illustrated tutorial, its worst mistakes are often long-range ones. A sword destroyed in episode two reappears in episode nine. A character's red coat turns blue three panels later. A recipe photo shows a frosted cake before the frosting step. Automated evaluators, usually a strong model prompted to grade outputs (the "LLM-as-a-judge" approach), are poor at catching these errors. They grade the whole sequence at once and get distracted by fluent prose or attractive images. This proposal turns the judge into a retrieval system. For every new segment of a generated sequence, the judge first writes retrospective questions: what earlier fact must be true for this segment to make sense? It then retrieves the earlier segments that should answer each question and checks that one assumption against that evidence. For images, the check is a targeted visual question about the specific earlier frame. The result is a judge that names the broken step, names the earlier step it contradicts, and assigns a confidence to that link.

The core claim is testable. A judge whose checks are driven by question-based retrieval detects and localizes continuity errors better than a judge that reads the full sequence, and its advantage should grow with the distance between an error and the earlier step it contradicts. At short distances the two should be roughly tied. If they are not tied there, something other than retrieval is producing the gain, and we will treat that as a sign the claim is wrong.

What exists and what is missing

Three bodies of work supply the pieces, but none puts them together.

The retrieval mechanism comes from Graph Representation of Narrative Context: Coherence Dependency via Retrospective Questions. That paper (NARCO) links passages of a novel with LLM-generated questions that a later passage raises and only an earlier passage answers. Indexing those questions improved recap identification by up to 4.7 F1@5 points over a GPT-4 baseline and raised long-document QA accuracy by 2 to 5%. NARCO, however, reads finished human text offline and never judges generated text. We reuse its question-as-edge idea as the search step inside a judge.

The evidence that judges need structure comes from two papers. AgentEval scores each step of an agent's tool-use trace along a dependency graph. It raised failure-detection recall from 0.41 (end-to-end grading) to 0.89 and found that 63% of step failures propagate from upstream steps. Interleaved Scene Graphs (ISG) breaks interleaved text-image grading into structural, holistic, block-level and image-specific checks. It reached a Pearson correlation of up to 0.907 with human raters and documented that unstructured multimodal judges inflate scores for attractive images.

Both papers fix their structure in advance. AgentEval reads it from the agent's tool calls. ISG checks a text block against its adjacent image. In a story, the dependencies are not given: the step a sentence relies on may be eight episodes back and share no words with it. Inferring those dependencies at judging time is the missing piece.

A third line of work, on keeping generated stories coherent, constrains the design. SCORE tracks item states and retrieves past episodes by embedding similarity while generating. It reaches about 98% item-status accuracy and raises GPT-4 complex-QA from 82.34 to 89.45. Its authors note, though, that similarity search misses causal links without shared vocabulary, and its evaluation grades episodes flatly. On the multimodal side, MRAMG-Bench and RAG-IGBench show that generators badly misorder procedural images. Both measure this with flat reference metrics such as edit distance and Kendall rank correlation, which say that order is wrong but not which step's assumption failed.

MRAMG-Bench also reports a tension the design must respect: text captions of images beat raw images for planning, because models saturate on many image inputs. ISG shows the opposite pressure, that only targeted checks on the actual pixels catch visual contradictions. Our judge resolves this split by role. It uses captions for the cheap search step and raw images only for the narrow verification step.

Core mechanism

Given a sequence of segments $s_1, \dots, s_n$, each a text passage, an image, or a text-image block, the judge processes each segment $s_j$ in four steps.

First, the judge model generates a small set of retrospective questions $Q_j$. Each question asks about a precondition that $s_j$ takes for granted, for example "Was the lid already removed?" or "What color was Maya's coat when she left the house?"

Second, each question is used as a query against an index of the earlier segments $s_{<j}$, represented by their text and by captions of their images. The search returns the earlier segment most likely to answer the question.

Third, the judge answers the question twice: once from the retrieved segment (asking a visual question of the raw image when the evidence is pictorial) and once from $s_j$. It labels the pair as supported, contradicted, or unresolved. An unresolved label means the precondition was never established, which is how omissions surface.

Fourth, the contradicted and unresolved pairs become directed edges in an error graph. Following AgentEval, an error is attributed to its earliest root, so that one bad step is not counted again in every downstream step.

The output is a localized report rather than a single Likert score. The story-level score is the fraction of preconditions that are supported. Because the judge reads one short evidence pair at a time, cost grows with the number of segments times the number of questions, not with the full context length.

Research program

The first piece of work is a test set with known answers. Long-range continuity errors are rare and costly to annotate in natural outputs, so we inject them into coherent sequences, at known positions and known distances from the earlier step they contradict. The injection types follow the failure taxonomy in Zooming from Context to Cue, which identifies omitted frames, details attributed to the wrong frame, and misread local cues. We add three types: state contradictions (an item SCORE would track as destroyed comes back), attribute drift (a character's appearance changes), and procedural reordering.

Source sequences come from four places. SCORE's 1,000 multi-episode stories (10 to 15 episodes each) cover long text. CoMM documents (CoMM), already filtered for textual and visual coherence, cover long procedural image-text sequences. The Lifestyle tier of MRAMG-Bench covers procedural ordering. VIST five-image stories (Visual Storytelling with Question-Answer Plans) provide a short-distance control.

Injected errors may carry artifacts that a judge could exploit, so we also collect a natural-error set. We generate stories and tutorials with an ISG-AGENT-style compositional pipeline and with an unguided model, and humans mark each continuity break together with the earlier segment it contradicts.

The second piece of work is the analysis. We measure how many injected antecedents the retrospective questions actually target. This number sets a ceiling on the method, and a low value would mean question generation, not retrieval, is the bottleneck. We break detection and localization down by antecedent distance, by error type, and by modality (text-only, image-only, cross-modal).

The third piece is a downstream use, kept secondary. We feed the localized reports back to a generator as revision instructions, in the spirit of GROVE's "asking-why" revision loop (GROVE). We then ask whether consistency, as measured by SCORE's own evaluation, improves more than with flat critique.

The judge backbone is held fixed across all variants. We use an open model (Qwen2-VL family) for most runs and GPT-4o for one replication, so that differences come from evidence selection rather than model strength.

Baselines

The do-nothing reference is reference-based scoring of the kind RAG-IGBench uses: ROUGE-1 for text and Kendall rank correlation and edit distance for image order, against the uncorrupted original. It shows how much of each error type surface metrics already detect. It cannot localize anything.

The strongest competitor is a full-context judge: the same backbone receives the entire sequence plus an error checklist and is asked to list every continuity break with its earlier cause. This is the obvious thing a practitioner would do, and on short sequences it should be hard to beat. To keep the comparison fair, it gets the same backbone, the same per-sample prompt-tuning budget on a held-out development split, and at least as many total input tokens as our method uses. The images are given raw or as captions, whichever scores better on development data. No reported result was found on Lacuna for a full-context judge on injected long-range continuity errors, so this baseline will be measured here. Beating it means higher localization accuracy (correct broken segment and correct earlier segment) at antecedent distances beyond about five segments, without losing at distances of one or two.

The strongest existing structured judge is ISG's four-tier evaluator. On ISG-Bench it reports a Pearson correlation of up to 0.907 with human annotators (Interleaved Scene Graphs). We run it unchanged on our sets and require our judge to stay within noise of ISG's human correlation on ISG-Bench's Visual Storytelling subset. That subset is mostly short and adjacent, which is ISG's home ground. Beating ISG is defined only on long-range errors, where its adjacency checks cannot reach.

A state-tracking judge adapted from SCORE extracts an item-status table episode by episode and flags illegal transitions. It is the strongest competitor on state contradictions in text, and we expect it to win or tie there. Our claim is that it misses preconditions that are not item states, such as motives, attributes, spatial setup and procedural prerequisites, and is blind to visual drift.

Three ablations isolate the key ingredient. The first replaces retrospective questions with dense-embedding search over earlier segments, using the new segment itself as the query. This tests whether the questions, not merely retrieval, do the work, which NARCO's results suggest. The second performs verification from captions alone, never looking at the retrieved image. This tests whether the visual question step is needed despite the captions-win finding. The third removes root attribution and simply counts every flagged pair, to measure double counting.

Why not just…?

Why not just give a long-context judge the whole story? This is the real threat. If frontier context windows handle 12-episode stories well, our method adds cost without benefit. The evidence against it is indirect. In the context-to-cue paper, base multimodal models described only 10.3% of images in long multi-image inputs and had a CHAIR object-hallucination rate of 50.6 in 8-image settings. SCORE also shows that GPT-4 without explicit memory loses complex-QA accuracy over long stories. Neither result measures judging directly, so the distance-stratified comparison settles it: if the full-context judge matches us at large distances, the core claim fails and we will report that.

Why not just track states explicitly, as SCORE does? For discrete items this likely works, and SCORE's 98% item-status accuracy shows it. The breakdown by error type tells us how much of continuity is not item status. If state tracking covers almost all natural errors in the human-marked set, the narrower and cheaper method should be preferred, and we will say so.

Why not just retrieve earlier segments by embedding similarity? Both SCORE's authors and NARCO report that similarity search misses causal links without shared wording. The dense-retrieval ablation measures that gap directly on generated, rather than human-written, narratives.

Why not just fine-tune a small detector on injected errors? A fine-tuned detector would be cheap. However, it would likely learn the fingerprints of our injection procedure, and it would not explain which earlier step was violated. We will train one as a supplementary comparison and evaluate it only on the natural-error set, where injection artifacts are absent.

Decisive checks

Four results decide whether the idea holds. The first and most important is localization accuracy plotted against antecedent distance for our judge and the full-context judge, on the SCORE and CoMM injected sets. We predict a gap that widens with distance and roughly closes at distances of one or two segments. A flat gap would suggest the gain comes from prompt structure rather than retrieval.

The second is agreement with human-marked natural errors, measured as Cohen's kappa against annotators so it can be compared with AgentEval's 0.84 expert agreement and SCORE's 0.78 annotator agreement. The third is the cross-modal slice: drift in visual attributes between non-adjacent frames, where the captions-only ablation should fail and the visual question step should recover the errors. The fourth is a cost check, total judge tokens per sample against the full-context baseline at equal or better accuracy.

If the first check fails, the proposal reduces to a cheaper way of running ISG-style evaluation. If the first passes but the third fails, the contribution is text-only, and multimodal retrieval for judging remains open.