Agents that work with people over many sessions have to keep track of facts that change: a budget gets revised, a price moves, a plan is replaced. Recent work shows that language-model agents handle this badly even with text alone. They "drift," answering from a superseded fact because the old fact was stated more often or more prominently, even when the correction is sitting in their context. In real workflows, though, corrections often come as pictures rather than sentences: a re-issued chart, a screenshot of a new price, a photographed receipt, an updated map. This proposal asks whether a correction delivered as an image is more easily ignored than the same correction delivered as text, after accounting for whether the model can read the image at all. It also tests a specific remedy. At the moment an image arrives, the agent should read it with questions about what it currently believes, not describe it in general terms.
The opportunity comes from two findings that have not been put together. Text-memory research shows that agents anchor to "loud" stale facts. Vision-language research shows that models which recognize every element of a picture still fail to reason over it, and that this failure grows over long sequences. If the two effects compound, then multimodal agents will act on obsolete information much more often than text benchmarks suggest. Text-based memory fixes would not catch this, because the failure would start at the point of perception.
Core Claim
We claim two things, each of which can be tested on its own. The first is diagnostic. Holding the scenario fixed, agents show a residual drift gap when an update arrives as an image instead of text, and the gap remains after restricting to cases where the model reads the image's value correctly when asked in isolation. Let $A_T$ be current-state accuracy when the superseding fact is delivered as text. Let $A_{I \mid r}$ be accuracy when the same fact is delivered as an image, restricted to items where an isolated readout of that image ("what is the value of X in this image?") is correct. The quantity of interest is
$$\Delta_{\text{res}} = A_T - A_{I \mid r},$$
which separates the cost of the update's modality from plain perception error. A positive $\Delta_{\text{res}}$ would mean that images are "quiet" evidence: the model sees them but does not let them overwrite memory.
The second claim concerns a method. When an image enters the conversation, the agent should first pull the list of state slots it currently tracks, such as "monthly rent" or "flight time." It should then ask the vision model, slot by slot, whether this image gives a new value, and from which region of the image. Only after that does it write structured, timestamped state entries. When an answer later depends on an image-sourced entry, the agent crops and re-reads that region before committing. We call this state-conditioned reading. We expect it to shrink $\Delta_{\text{res}}$ well beyond what generic captioning followed by a text memory system achieves.
Grounding in Prior Work
The motivating evidence comes from three places. Can Agent Memory Systems Track Evolving State? introduces StateMemBench (234 multi-session scenarios in Research, Shopping, and Personal Finance) and the StateMem system. StateMem parses each turn into state units carrying dependency links (derived_from, coupled_with) and runs a deterministic rechecker that flags downstream values gone stale. The paper also shows that drift persists with oracle context and that chain-of-thought can make it worse. All of it is text-only, and we reuse its state-unit and rechecker design as our memory backbone. MET-Bench shows the visual side of the problem: models identify individual shell-game swaps nearly perfectly yet fall to chance after about 20 visual swaps while staying near-perfect over 100+ textual swaps. Its game states are fully visible at every step, though, so nothing is superseded and nothing must be retrieved. MEMTRACK tracks state across Slack, Linear, and Gitea logs over simulated weeks, also text-only, and finds that adding memory tools does not reliably help frontier models.
Two findings constrain the design. Plot Twist shows that models read elementary chart values poorly and lean on titles: accuracy is up to 79% when the title contains the answer and falls to the 30% range otherwise. Generic descriptions of an image will therefore miss exactly the values that changed. This is why our reading step asks about specific slots and draws on the code-based extraction of Synthesize Step-by-Step. MATH-Vision attributes 42.2% of GPT-4V errors to reasoning after correct perception and finds chain-of-thought sometimes hurts. This warns us against relying on free-form reasoning traces and motivates the perception-controlled metric above. DCoT shows that separating "look at this region" from "reason about it" gives consistent gains, and our crop-and-re-read step borrows that separation.
The multimodal knowledge-editing line is the closest work on visual obsolescence. MMKE-Bench and CARML update a model's visual facts and evaluate the result with static question answering. Our setting differs in three ways: the knowledge being updated is the agent's episodic state rather than world knowledge in its weights, the updates arrive mid-conversation, and they carry dependencies that propagate. On the agent side, RESPOND names "agents seeing flood images" as unaddressed future work, and we treat its setting as a secondary, illustrative application.
Research Program
The first component is a paired benchmark, MM-StateMemBench. We take the StateMemBench scenarios and, for each superseding update turn, render the same fact as an image of a type that fits the domain. Shopping updates become price screenshots, personal finance updates become revised bar charts or receipts, and research updates become figure revisions or table images. Every item therefore exists in a text-update version and an image-update version, with identical dialogue otherwise and identical closed-pool grading. For each image we also record an isolated readout question, which gives us the perception-conditioning set $r$. We vary three factors: rendering difficulty, using the perturbations Plot Twist found harmful (grid lines and marker style); how "loud" the stale fact is, measured by its repetition count; and whether the update has downstream dependents that must also change. A smaller second split uses MET-Bench-style sequences, inserting a hidden-state interval so the agent must remember rather than re-observe.
The second component is the agent. It uses StateMem's state units and rechecker, adds image provenance (image id, crop box, timestamp), and runs state-conditioned reading at ingest and targeted re-reading at answer time. Everything runs on a fixed open vision-language backbone plus one frontier model accessed through an API.
The third component is analysis. We estimate $\Delta_{\text{res}}$ for each model, domain, rendering difficulty, and loudness level. We test whether explicit reasoning traces increase or decrease drift in the multimodal setting. This resolves, at least in this domain, the conflict between StateMem's finding that chain-of-thought hurts and the date-controlled tool use finding that ReAct with chain-of-thought helps.
Baselines
The naive reference point is a native multimodal long-context agent that receives the entire session history, images included, and answers directly. It shows how much drift occurs when nothing is done. It runs on the same backbone, and its context is truncated only where the model's limits force it, which we report.
The strongest existing method is StateMem with StateMemWrapper from Can Agent Memory Systems Track Evolving State?. We feed it image turns either as dense captions or as native image inputs, whichever scores higher on a held-out tuning split. On the text-only StateMemBench, that paper reports current-state accuracy of 0.363 for StateMem against 0.205 for the strongest memory baseline with a DeepSeek-V4-Flash backbone, and wrapper lifts of +32 to +67 points across six memory backends. We do not yet have a number on our image-update split. No reported result exists on Lacuna for any multimodal state-drift benchmark, so we will rerun StateMem ourselves and first check that our reimplementation reproduces its text-split accuracy within noise. Beating it means a statistically significant gain on image-update items at a matched number of vision-model calls, with no loss on text-update items.
The third baseline is the multi-agent post-hoc correction system of Mitigating LVLM Hallucination via Multi-agent System, where a supervisor queries vision tools and debaters reconcile the results. We adapt it so that it verifies answers against the stored images. This is the strongest "check visual claims afterward" competitor.
The key ablation replaces state-conditioned reading with generic captioning, at the same call budget and with captions allowed to be as long as needed. It then removes the answer-time re-reading separately. Together these isolate whether the gain comes from conditioning perception on current beliefs or merely from spending more perception compute. Every system gets the same backbone, the same scenarios, the same tuning split, and a capped number of vision calls per image.
Why not just…?
Why not just caption every image and run the best text memory system? This is the most serious threat. If dense captions plus StateMem matches our method, then the method claim fails and the contribution reduces to the benchmark and the $\Delta_{\text{res}}$ measurement, which we would report as such. Our reason to expect captioning to fall short is that a caption is written without knowing which values matter. Plot Twist shows that elementary values such as axis readings and minima are exactly what models get wrong unless something forces attention onto them. GeoSQA adds that even perfect human transcription does not by itself fix reasoning. The captioning ablation at matched call budget settles the question directly.
Why not just keep all images in a long context window? StateMem's own experiments show that drift persists with oracle context, and MET-Bench shows that visual sequences degrade faster than textual ones. On the other hand, the strongest frontier models (Gemini 2.5 Pro was MET-Bench's exception) may tolerate long visual histories. We include a frontier long-context baseline and report the residual gap per model, rather than assuming the gap holds universally.
Why not just prompt the agent to trace the value chain, or to think step by step? StateMemWrapper's value-chain prompt is cheap and lifted text accuracy by 32 to 67 points. We therefore run it with native images as a baseline in its own right. Our hypothesis is that it fails specifically when the newest link in the chain sits inside an image the model glances past. The paired design tests this precisely: the wrapper should close the gap on text updates but not on image updates.
Why not edit the model whenever a visual fact changes? Episodic state such as "my rent is now this amount" belongs to the user's session, not to world knowledge, so editing weights is the wrong layer. The evidence also points against it. MEND falls below 1% reliability after 1,000 sequential edits in CARML, and MMKE-Bench finds that edits fail to carry through to multi-hop use. We therefore do not run weight editing as a main baseline. We do include CARML-style retrieval of edits as a sanity check on the dependency split.
Proposed Experiments and Decisive Checks
The first decisive check is whether $\Delta_{\text{res}}$ is reliably positive on MM-StateMemBench, measured with bootstrap confidence intervals over scenarios. If it is near zero, image-update drift is just perception error, and the right fix is better chart reading rather than anything memory-specific. We would then report that negative result.
The second check is whether $\Delta_{\text{res}}$ grows with how loud the stale fact is. That would confirm a compounding of salience-driven drift and the multimodal reasoning gap, as opposed to two independent error sources.
The third check is the captioning ablation at matched call budget, which decides whether the method claim survives. Secondary measurements include accuracy on dependents that must change after an image update, which tests whether the rechecker propagates image-sourced changes, and the rate of false stale flags caused by misreads. The second measurement probes the concern raised by Sheaf-Guarded Updates that noisy loaders corrupt strict consistency checks.
Finally, as a small illustrative case, we build a RESPOND-style evacuation scenario in which a later flood-map image contradicts an earlier tweet that a bridge is open. We measure how often agents route over the closed bridge with and without state-conditioned reading.
References
- Can Agent Memory Systems Track Evolving State?
- MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models
- MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
- Reliable Lifelong Multimodal Editing: Conflict-Aware Retrieval Meets Multi-Level Guidance
- RESPOND: Realistic Environment Simulation of Population and Natural Disasters with LLM-Driven Agents
- Measuring temporal effects of agent knowledge by date-controlled tool use
- Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent System
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- Sheaf-Guarded Updates: Streaming Structural Verification for Evolving Agent State