Agents that watch hours of video usually remember it by writing text notes about what they saw and later reasoning only over those notes. That is cheap, and graph-structured note stores now beat feeding a model the whole history. The cost is that the notes cannot be checked. A misread gesture or an over-eager guess about a character's motive becomes a permanent "fact." Every later deduction inherits it, and the frames that could correct it are never looked at again. This proposal builds a memory in which every remembered claim carries two things: a pointer back to the frames it came from, and a label saying whether it was directly observed or inferred. When the agent answers a question, it follows cause-and-effect and character links through the memory to find the few claims its answer depends on. It then looks again at exactly those frames to confirm or revise them before committing.
The two-sentence version is this. Long-video models fail at story-level reasoning (counterfactuals, motivation tracking, open-ended plot reconstruction) partly because their memories are lossy and unrevisable, not only because they reason badly. Linking each memory to its visual evidence, and letting the structure of the story decide which evidence to re-inspect, should repair much of that failure at a fixed viewing budget. The project also measures how much of the failure is left once memory is perfect.
Why this, and why now
The motivating evidence comes from three places. Diagnostic benchmarks show models recognize surface events well but lose the thread of a story. On SocialReasonBench, which builds ground-truth counterfactuals from branching game narratives, Gemini 3 Pro and GPT-4o reach about 84–85% on intent recognition but only 62% and 50% on counterfactual reasoning. On MultivationBench, which asks models to track characters' motivations across illustrated stories, the best model is consistent across a full story only 0.80% of the time. Accuracy there decays past about 13 frames, and the dominant error is over-interpretation, meaning false-positive attributions. NARU finds that sequence tracking on up to four-hour videos collapses when multiple-choice options are removed. The options were acting as scaffolding.
The closest system is GCAgent. It organizes long video into an episodic memory of roles, places and narrative causal links ("conflict", "resolution") and lifts Qwen2.5-VL 7B to 73.4% on the Video-MME long split. Its memory, however, is built by a language model reading transcripts and captions, and it degrades under translation distortion. That is the failure our design targets: once a caption is written, the pixels are gone. LogiShot supplies independent evidence that decoded captions drop logically relevant visual information. In cross-shot video generation, conditioning on a vision-language model's hidden states instead of its captions improved logical correctness. Text-only memory architectures supply the structure we reuse. MAGMA splits memory into temporal, causal, semantic and entity graphs and reaches an LLM-judge score of 0.700 on LoCoMo against 0.481 for full context. LightMem defers cleanup to offline consolidation. Neither touches perception.
Two results constrain the design. Plot Twist shows that chart reasoning hinges on small visual details and that models quietly substitute textual priors when those details are hard to read. Any compression that throws away pixels is therefore suspect, which argues for keeping the frames on disk rather than summarizing them away. Before It Persists separates untrusted "evidence" from trusted "belief" at write time. That separation nearly eliminates memory poisoning, but it lowers benign task completion from 0.655 to 0.418. Our observed/inferred split borrows the idea, and we must measure whether it over-quarantines legitimate inferences in the same way.
Core Claim
A narrative memory whose claims are typed (observed versus inferred) and linked to source frames gives better story-level reasoning over long video than either text-only structured memory or similarity-based frame retrieval, at an equal budget of frames re-read per question. The memory decides which frames to re-inspect by following the causal and entity links relevant to the question, rather than by visual or textual similarity to it. That choice is what matters. Counterfactual and motivation questions depend on evidence that does not resemble the question: the cause of a betrayal looks nothing like the betrayal. The predicted gains are therefore specific. They should be large on counterfactual, causal-antecedent and motivation-consistency items and near zero on perception-easy items such as intent recognition.
Research Program
The system has a write phase and a read phase, both run on one open backbone, Qwen2.5-VL 7B, chosen so the comparison with GCAgent's reported numbers is direct.
Writing. The video is processed in segments. For each segment the model writes atomic claims about entities, events, states and motivations. Each claim carries a timestamp range, a pointer to the frames (and, where possible, regions) it rests on, and a type. Observed claims describe something visible or audible. Inferred claims are interpretations: an intent, a motive, a relationship change. Edges follow MAGMA's split into temporal order, cause-and-effect, and shared entity. An inferred claim must link to the observed claims that support it. It stays tentative until at least two independent observed claims support it and none contradicts it. This promotion rule targets MultivationBench's over-interpretation directly. The frames themselves are stored outside the prompt and cost no tokens until they are re-read. Offline consolidation in the LightMem style merges duplicate entities between segments.
Reading. Given a question, the model selects starting claims, then walks causal and entity edges to assemble a candidate reasoning chain. For counterfactual questions, the walk goes forward from the hypothetically changed event to find what depends on it. The model then spends a fixed budget of $B$ re-read frames on the claims in that chain, ranked by how much the answer depends on them and how uncertain they are. For each, it asks a targeted verification question of the original frames, for example "does this frame show her pocketing the key?". Claims that fail verification are demoted or rewritten, the chain is re-walked, and the model answers from the verified chain. Revisions are written back, so the memory improves with use.
Analysis. The analytic half of the project separates memory error from reasoning error, borrowing the logic of GeoSQA, which replaced diagrams with perfect human transcriptions and found models still failed. On a subset of SocialReasonBench and MultivationBench, annotators will correct the written memory claim by claim, producing an oracle memory. The difference between oracle-memory accuracy and normal-memory accuracy measures how much failure comes from bad memory. That difference bounds what any memory fix, including ours, can recover. The gap between oracle-memory accuracy and human accuracy is the reasoning gap that remains.
Baselines
The naive reference point is the backbone given uniformly sampled frames up to its context limit, with no memory. This is the "do nothing" condition, and it establishes how much any structure helps. It runs on the same backbone with the same total frame count the full system touches across writing and reading.
The strongest existing method in this regime is GCAgent with the same Qwen2.5-VL 7B backbone. It is the strongest competitor because it already combines structured episodic memory with narrative causal links over long video. That makes it the only prior system that tests whether evidence linking, rather than structure, is what helps. Its reported result is 73.4% accuracy on the Video-MME long split (GCAgent), against 72.1% for GPT-4o. We will reproduce GCAgent from its description on the same videos with matched model-call budgets at write time. Beating it means a statistically reliable gain on SocialReasonBench counterfactual and causal-antecedent items and on MultivationBench consistency, without falling below 73.4% on Video-MME long. A Video-MME tie with gains on the narrative benchmarks would still support the claim, since Video-MME is not built around counterfactuals.
The proprietary long-context reference is Gemini-class full-video ingestion. Reported figures are 76.2% on NARU for Gemini-3-Flash (NARU) and 62% on SocialReasonBench counterfactuals for Gemini 3 Pro (SocialReasonBench). We will not claim to beat these with a 7B model. The target is to close a measurable fraction of the gap between open models (29.6–39.8% on NARU) and these numbers. Where API access permits, we also run our memory with a proprietary backbone against its own full-context version.
Three ablations isolate the key ingredient. The first, text-only typed memory, removes re-reading. The second, similarity re-reading, keeps the frame budget $B$ but selects frames by embedding similarity to the question instead of by graph walk. The third, no type split, drops the observed/inferred distinction and the promotion rule. All variants share the backbone, the stored memory where applicable, and the per-question frame budget, and we sweep $B$ for all of them.
Why not just…?
Why not just feed the whole video into a long-context model? For proprietary models this is a real competitor, and on NARU it is currently the best number available. The evidence that it falls short on narrative structure is partial. On Video-MME, Gemini 1.5 Pro loses 14.3 points going from short to long videos (Video-MME). Full context scores 0.481 against MAGMA's 0.700 on text LoCoMo. NARU shows that even the strong models' sequence tracking depends on multiple-choice scaffolds. We will settle this with an open-ended NARU variant, where answers must reconstruct the event sequence in free text and are scored against the gold sequence. Our prediction is that the gap between multiple-choice and open-ended accuracy shrinks more under our memory than under full context.
Why not just retrieve raw frames by similarity and let the model look again? This is the most serious threat, because it captures the "keep the pixels" half of the idea without any graph. If similarity re-reading matches graph-guided re-reading at equal $B$ on counterfactual and causal items, the graph is unnecessary, and the contribution reduces to "store frames and re-read them." We will report that outcome as such. The specific prediction is a divergence by question type. On intent recognition the two should tie, because the evidence resembles the question. On causal-antecedent and counterfactual items, graph-guided selection should win, because the evidence is linked to the question causally rather than by resemblance. A per-type breakdown on SocialReasonBench's seven dimensions decides this.
Why not just prompt the model to reason step by step over its notes? MATH-Vision reports that chain-of-thought had inconsistent effects on multimodal models and lowered accuracy for some, because intermediate steps compound perceptual mistakes. That is exactly what unverifiable notes would invite. We include chain-of-thought over GCAgent-style memory as a baseline. The oracle-memory experiment then shows whether step-by-step reasoning is limited by the notes or by the reasoning itself.
Why not just filter bad memories at retrieval time instead of typing them at write time? Before It Persists shows retrieval-time filtering leaves contaminated memory in place. Its write-time remedy over-quarantines, however, so we treat the promotion rule as a hypothesis, not a given. The no-type-split ablation and a recall measure on true inferred claims will show whether tentative status suppresses correct motivations along with false ones.
Proposed Experiments
The evaluation uses SocialReasonBench (532 items, scored per dimension), MultivationBench (exact match, full-story consistency, and false-positive rate on motivation attributions), NARU in both its multiple-choice form and our open-ended variant, and the Video-MME long split as a check that general long-video accuracy does not regress. LongVideoBench is used by GCAgent, but no reported result was found on Lacuna for it, so it serves only as a secondary within-study comparison. Beyond task accuracy, we audit memory quality directly. On sampled claims, annotators judge each claim against its linked frames, giving the fraction of observed and inferred claims that are correct, before and after read-time revision.
Four checks decide the project.
The first is the per-type comparison between graph-guided and similarity re-reading at matched frame budgets. A win concentrated on counterfactual and causal items supports the core claim. A tie means the graph is unnecessary.
The second is whether MultivationBench's false-positive rate falls under the promotion rule while true-inference recall stays within a few points of the no-type-split ablation. This tests whether typing fixes over-interpretation without the over-quarantining seen in SAGE-Mem.
The third is the oracle-memory decomposition. If perfect memory adds little over normal memory, the bottleneck is reasoning, and the project's conclusion becomes that the reasoning gap persists even with clean multimodal memory. That would be a useful negative result. If perfect memory adds a lot, we report what fraction of that gap evidence-linked re-reading recovers.
The fourth is the accuracy–budget curve as $B$ grows from zero, where zero is text-only memory. If the curve flattens after a handful of frames, targeted re-perception is cheap enough to deploy.
References
- SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos
- MultivationBench: A Benchmark for Multimodal Sequential Motivation Reasoning
- NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video
- GCAgent: Long-Video Understanding via Schematic and Narrative Episodic Memory
- LogiShot: Logically Coherent Cross-Shot Video Generation
- MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents
- LightMem: Lightweight and Efficient Memory-Augmented Generation
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Before It Persists: Write-Time Defense for Multimodal Agent Memory
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset