An assistant that watches someone cook or repair something must keep answering one plain question: what state is each object in right now? Two things make this hard. First, most state changes are never stated or clearly shown. The egg gets whisked off-camera, or the flour becomes "dough" without anyone naming it. Second, the two main evidence streams disagree about when things happen. A narrator says "now I crack the eggs" several seconds before or after the hands actually do it. So when the system holds a visual observation ("egg: raw" at 1:02) and a narrated action that would change it ("whisk the eggs," spoken at 1:00), it cannot tell whether the observation was overwritten, is about to be overwritten, or is still current. Current systems settle this by committing to a single alignment and then trusting whatever came last. Near the moments that matter, that rule picks a stale state.
We propose to treat every remembered object state as a belief that expires. The belief stays valid until some action that plausibly touched that object plausibly happened after it was observed. The probability of expiry is computed by averaging over uncertainty in when each narrated action really occurred, and over uncertainty in which object an under-specified action affected. In two sentences: knowledge obsolescence, usually framed as facts going stale with calendar time, here becomes object states going stale because of actions whose timing relative to the camera is uncertain. Getting the "is this still true?" decision right requires keeping that timing uncertainty rather than discarding it.
Core Claim
The idea rests on three claims. The first is that a large share of current-state errors in narrated procedural video are supersession errors: the model reports a state that was true earlier. The second is that these errors concentrate in windows where a narrated state-changing action and a visual state observation fall within the typical narration offset of each other. The third is that a model which carries a posterior over each action's true visual time will cut those errors substantially, relative to the same pipeline using the single best alignment. The posterior is a probability distribution, learned from data, over where in the video each sentence's action actually happened.
If a single best alignment, or even a fixed global time shift, performs as well, then the claim is false and the alignment-uncertainty component should be dropped. The experiments are built to detect exactly that outcome.
Where This Sits
Each direction supplies one piece, and none supplies the combination.
From knowledge obsolescence, two papers motivate the proposal. Can Agent Memory Systems Track Evolving State? shows that language-model agents keep choosing superseded values even when the current value sits in the prompt. It also reports that step-by-step reasoning can make this worse. Its StateMem system fixes the problem with a dependency graph that invalidates derived facts when a base fact changes. That design assumes clean, totally ordered text turns, which is precisely what narrated video lacks. Mitigating Temporal Misalignment by Discarding Outdated Facts predicts how long a fact lasts and lowers confidence once the gap between query time and evidence time exceeds that lifespan. We borrow its survival-curve idea for passive changes such as dough resting or ice melting. Our expiry is driven mainly by inferred actions, not elapsed time.
From cross-modal misalignment, we take the alignment machinery. Learning to Ground Instructional Articles in Videos through Narrations (VINA) aligns steps to frames through both a direct and a narration-mediated pathway, reaching 66.5% R@1 on HTM-Align. It outputs a localization, though, not a distribution, and it never asks what the localization implies for object state. EgoTV shows that constrained dynamic-programming alignment beats soft attention by a wide margin: 90.0 vs 56.2 F1 on Novel Tasks. That alignment is strictly monotonic, so it breaks when narration runs out of order. Our per-action posterior keeps only a soft ordering preference. EM^2Mem organizes long video into fixed 30-second event cells that bind captions and transcripts. A 30-second window is coarser than the offsets we care about, and its cells index what was shown rather than what changed.
From implicit state inference, we take the action and state extractors. Predicting Implicit Arguments in Procedural Video Instructions recovers the omitted object or tool of an action, such as the unnamed bowl in "whisk until fluffy." That is how we decide which object an action might have changed. Learning Multiple Object States from Actions via Large Language Models turns narrated actions into cumulative state pseudo-labels on the MOST dataset. It is the closest existing work and our main point of contrast. It assumes the narration's order and timing translate directly into forward state changes, with no notion of a state being overwritten under timing uncertainty. Do Pre-trained Vision-Language Models Encode Object States? constrains the design. Off-the-shelf image-text models fall to 40.8% when choosing between states of the same object, and cropping the object does not help. Frame-level state evidence is therefore weak, so the system must lean on action evidence and must know when that evidence has expired.
Research Program
The system has three parts.
The first part extracts events with timestamps. From the automatic speech transcript, an implicit-argument parser in the style of iSRL-Qwen2-VL produces action tuples of the form verb, object, location/tool. Each tuple carries a probability distribution over which tracked object it affected, including objects that were never named. From the video, a per-frame multi-label state classifier produces noisy state observations. This is the self-trained student from the MOST work.
The second part is an alignment posterior. For each narrated action, we combine two ingredients. One is VINA-style text-to-frame similarity. The other is a learned prior on narration offset, fit on HTM-Align and HT-Step, where ground-truth alignments exist. The prior is allowed to be asymmetric and to depend on the verb. Together these give a distribution $p(\tau_a)$ over the action's true time $\tau_a$.
The third part is the expiry computation. A visual observation that object $o$ is in state $s$ at time $t_v$ stays valid at query time $q$ with probability
$$ V_s(q) = S_s(q - t_v)\prod_{a}\Big(1 - \pi_a(o)\,\Pr[t_v < \tau_a \le q]\Big). $$
Here $\pi_a(o)$ is the probability that action $a$ changes object $o$ (from the implicit-argument parser), $\Pr[t_v < \tau_a \le q]$ comes from the alignment posterior, and $S_s$ is a survival curve for passive change of state $s$ (lifespan-style, fit per state type). The product form assumes the actions act independently. Expired mass is passed to the action's predicted outcome state, so "raw" decays into "whisked" rather than into nothing.
When no state's validity clears a threshold, the system abstains and re-inspects frames near $q$. This mirrors how Zhang and Choi trigger retrieval only when confidence falls.
The analysis targets the reasoning paradox in the file. Structured intermediate outputs help implicit-argument recovery, yet free-form reasoning hurts under supersession. We test whether forcing a language-model backbone to write out the expiry quantities above, instead of a free-form chain of thought, gets the first benefit without the second.
Baselines
The naive reference point is a last-writer-wins pipeline with no alignment at all. Narrated actions are placed at their speech timestamps, and the state is whatever the most recent evidence implies, under the rule of inertia from ProPara's ProLocal: an entity stays in its last known state unless an action changes it.
The strongest competitor is the same pipeline with a strong alignment step. VINA localizes each narrated action, the MOST-style state classifier supplies visual observations, inertia supplies the rule, and the implicit-argument parser is shared. This is the strongest competitor because it differs from our system in exactly one respect: it uses the single best alignment instead of the posterior. Our ablation of "argmax $\tau_a$ in the equation above" is therefore nearly the same system. Running both makes the comparison clean. Beating this baseline means a statistically significant gain in current-state accuracy on near-conflict queries (defined in the next section), with no loss on far-from-conflict queries.
The second strong competitor is a long-context video-language model. It receives subsampled frames plus the full timestamped transcript and is asked the current-state question directly, both with and without the value-chain prompt of StateMemWrapper. That prompt lifted six memory backends by 32 to 67 points on StateMemBench, though on text only. All systems use the same video-language backbone family, and every prompt and threshold gets the same tuning budget, a fixed number of validation runs.
The third baseline is StateMem itself, run over a merged text timeline of transcript lines and frame captions, to test whether a text-only dependency graph is enough. Its reported result is 0.363 current-state accuracy versus 0.205 for the strongest baseline on DeepSeek-V4-Flash, on StateMemBench, in Can Agent Memory Systems Track Evolving State?. That result is on dialogue, not video. For the benchmark we build there is no reported result found on Lacuna, and for MOST the file reports only a relative gain of over 29% mAP, not an absolute figure.
Ablations remove one ingredient at a time: the alignment posterior (argmax instead), the implicit-argument term (only explicitly named objects can be changed), and the survival term (no passive expiry).
Why not just…?
Why not just shift the narration by a learned global offset and keep last-writer-wins? This is the real threat. If offsets are nearly constant within a video, one shift fixes most conflicts and the posterior adds nothing. We will fit the best global shift and the best per-video shift on HTM-Align, report the measured spread of offsets around them, and include both as baselines. If either matches our model on near-conflict queries, we conclude that offsets are predictable enough to drop the uncertainty machinery, and we report that as the result.
Why not just give a long-context video-language model everything? The file gives two reasons to expect this to fail at supersession specifically. Models given oracle context still anchor on stale values (Can Agent Memory Systems Track Evolving State?). GPT-4o forgets implicit state changes such as pork being spiced (Predicting Implicit Arguments in Procedural Video Instructions). That evidence is all text or clip-level, however, so we run this baseline directly and break down its errors into stale versus other.
Why not just fine-tune an end-to-end video model on current-state queries? Soft attention lost badly to explicit alignment on EgoTV, and the cross-modal fusion literature in the file reports visual bias under unconstrained attention. A fine-tuned model may still learn the offset statistics. We include one fine-tuned variant trained on the same training queries. It is evaluated on held-out object categories, where memorized offset patterns should not transfer.
Proposed Experiments
We build a current-state query set on MOST, whose six object categories carry frame-level multi-label state annotations and narrations. Each query asks for the state of object $o$ at time $q$ and is answered from the ground-truth frame labels. Queries are sampled so that half fall within the offset window of a narrated action affecting $o$. We call these near-conflict queries. The other half fall far from any such action.
To test robustness, we add a controlled version: transcript timestamps are perturbed by offsets drawn from the HTM-Align empirical distribution at increasing scale. We also mine verbal corrections in the transcripts, utterances containing "actually" or "instead" followed by a changed object, and form a small retraction subset. Those annotations would be new and are counted as such.
The metrics are current-state accuracy, stale-answer rate, and expected calibration error of $V_s(q)$. The stale-answer rate follows StateMemBench's closed-pool grading and is the fraction of answers naming a state that held earlier. We also report standard MOST mAP to confirm frame-level segmentation does not regress. A secondary transfer test applies the same queries to EgoLifeQA-style long recordings, where the survival term should matter most.
Three checks are decisive. The first is whether the posterior beats the argmax ablation on near-conflict queries while tying on far queries. The second is whether that gap grows with injected offset scale; a flat gap would mean alignment uncertainty is not the operative cause. The third is whether the stale-answer rate, not overall error, is what falls. If the gain instead comes from the implicit-argument term alone, the contribution is better object attribution, not offset-aware expiry, and we will say so.
References
- Can Agent Memory Systems Track Evolving State?
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- Learning to Ground Instructional Articles in Videos through Narrations
- EgoTV: Egocentric Task Verification from Natural Language Task Descriptions
- EM^2Mem: Event-Centric Multimodal Memory for Large Language Models
- Predicting Implicit Arguments in Procedural Video Instructions
- Learning Multiple Object States from Actions via Large Language Models
- Do Pre-trained Vision-Language Models Encode Object States?
- Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension