Tracking What Happened to the Onion: Per-Entity Latent States for Filling Omitted Event Arguments

Procedural text and video leave things out. A recipe says "chop the onion, then add to the pan, then stir." "Add" and "stir" never name the onion or say where it ends up, because the writer assumes you remember. Current systems fill these gaps by generating tokens from attention over the history. Even strong vision-language models forget which ingredient was already oiled, chopped, or moved (Predicting Implicit Arguments in Procedural Video Instructions). We propose to give each entity in a procedure its own probabilistic latent state: a small vector the model keeps, updates each time a structured event acts on that entity, and lets drift with growing uncertainty while the entity goes unmentioned. An omitted argument then becomes an inference question with a proper answer: given every entity's current belief state, which one most plausibly fills the empty "what" or "where/with" slot of this verb, and how sure are we?

This connects two literatures that have not met. Recurrent latent state models, meaning sequence models whose hidden variable at step $t$ is drawn from a learned prior that depends on the previous step's latent, are well developed for continuous signals and raw text. Structured event representations, which break an event into a predicate plus labeled role fillers, are well developed for narrative and procedural understanding, but they are always paired with deterministic sequence encoders. The new element here is that the structured frame decides which latent state gets updated and how. The latent variable, in turn, supplies a calibrated distribution over which entity is bound to each role. That binding is a discrete choice, and it is exactly what current event models get wrong under ellipsis.

Core Claim

A state-space model whose latent state is factored by entity, and whose transitions are driven by predicate-argument frames, will fill implicit arguments more accurately than a fine-tuned autoregressive vision-language model when the entity was last mentioned several steps earlier. It will also give well-calibrated confidence over candidate fillers where the deterministic alternatives do not. We make two claims and test them separately. The first is that entity-indexed state helps accuracy at long gaps. The second is that making that state stochastic improves calibration and handling of truly ambiguous slots. We expect a deterministic entity memory to capture much of the first gain, so the second claim has to earn its place through calibration and ambiguity metrics, not F1 alone.

Relation to Prior Work

Several papers motivate the gap. Predicting Implicit Arguments in Procedural Video Instructions turns cooking steps into {verb, what, where/with} tuples, releases the Implicit-VidSRL benchmark, and fine-tunes Qwen2-VL to predict missing arguments. It reports that predicting structured tuples before text raises implicit-argument F1 by 31.8%, while GPT-4o loses track of entity state changes. That result tells us structure helps. Our proposal keeps the tuple representation but replaces token-level recall with an explicit belief state per entity, which that paper does not have.

Rich Event Modeling for Script Event Prediction enriches narrative events with Abstract Meaning Representation roles and verb senses. Semantic and Sentiment Dual-Enhanced Generative Model for Script Event Prediction injects parsed semantic graphs into BART. Both show that role structure matters, since removing the semantic aggregator costs SSD-GM 2.59 points on the Multiple Choice Narrative Cloze task. Both run the event sequence through deterministic attention, and neither tracks individual entities as evolving states. Prompt2Gaussia is the closest probabilistic work. It uses Gaussian prompt embeddings whose variance reflects ambiguity and gains about one point from the variance term. Its distribution is computed once over the whole context rather than propagated step to step, and it breaks when a new participant appears. We propagate per-entity distributions through time and allow new entities to be created explicitly.

Other papers supply the method. Deep Recurrent Generative Decoder for Abstractive Text Summarization conditions a latent $z_t$ on $z_{t-1}$ and a GRU state, and it reports that this latent track preserves "who did what to whom" structure. Our transition prior borrows that design, but DRGD keeps one global latent over surface tokens, with no roles and no entities. That makes a DRGD-style single latent the natural ablation for our entity factoring. Probabilistic Recurrent State-Space Models shows that unrolled variational inference with a recognition model for the initial state tracks hidden physical state from partial observations. We reuse the idea of an initial-state recognition network when an entity is first introduced. PR-SSM has never been applied to symbolic events, and we do not use its Gaussian-process transitions, which would scale poorly to learned text and vision embeddings.

One paper constrains the design. ROAD finds that multiplying independent agent, action, and location marginals matches joint modeling for road events. This warns us that joint role modeling is not automatically better. So the binding score couples entity state with verb and role, but the roles of one frame are scored conditionally independently given the entity states. Whether that coupling is needed is tested directly in the ablations.

Model

A procedure is a sequence of frames $f_1,\dots,f_T$. Each frame has a verb and role slots that may be filled, explicitly or implicitly, or empty. Entities $k=1,\dots,K_t$ are created when first mentioned or seen, and new ones can be spawned later. Each entity carries a latent state $z_t^k$.

The transition prior moves $z_{t-1}^k$ through a learned update conditioned on the previous frame's verb embedding and the role, if any, that entity $k$ held in it. Entities that were acted on change substantially. Entities that were not involved persist with a small learned noise term, so uncertainty about an unmentioned entity grows the longer it stays unmentioned. That growth is precisely the behavior a deterministic memory cannot express.

For each role $r$ in frame $t$, a discrete binding variable $b_t^r$ chooses among existing entities, a "new entity" option, and "none":

$$p(b_t^r = k \mid z_t^{1:K}, v_t) = \frac{\exp g(z_t^k, v_t, r)}{\sum_{j} \exp g(z_t^j, v_t, r)},$$

Here $v_t$ is the verb embedding and $g$ is a small network that scores how well entity $k$'s current state fits role $r$ of verb $v_t$. Explicit arguments are observations: their text span embedding and, when available, the visual region feature are emitted from the bound entity's state. Implicit arguments have no emission and are inferred.

Training maximizes an evidence lower bound. The bound uses an amortized recognition network over the continuous states and a small number of samples. The discrete bindings are enumerated exactly, since procedures involve only a handful of entities. At test time, the posterior over $b_t^r$ for an empty slot is the prediction, and its entropy is the confidence. Frames come from the silver SRL annotations of the Implicit-VidSRL training set, about 18K samples. Visual and text features come from the same frozen encoder used by the strongest baseline, which keeps the comparison about the state model rather than the features.

Research Program

The work has three parts. First, build the model above and a deterministic twin that is identical except that the states are point vectors, trained with the same binding objective. Second, analyze where each variant wins by bucketing test slots by the gap, in steps, since the correct entity was last explicit, and by whether annotators judge the slot ambiguous. For the ambiguity analysis, we will collect a small second-annotator layer over the verified Implicit-VidSRL frames, marking every entity an annotator would accept. Third, test transfer to text-only narrative chains on the Multiple Choice Narrative Cloze task. This check asks whether per-entity state helps when only one protagonist is tracked, and we expect little gain there. A null result on this task would be informative rather than damaging.

Baselines

The naive reference is a "most recent compatible entity" heuristic. It fills an empty "what" slot with the most recently mentioned patient, and an empty "where/with" slot with the most recently mentioned container or tool, using the same entity inventory our model sees. If our model cannot clearly beat this heuristic at gaps of three or more steps, the latent machinery is not justified.

The strongest existing method is iSRL-Qwen2-VL from Predicting Implicit Arguments in Procedural Video Instructions, a fine-tuned multimodal model that decodes SRL tuples autoregressively. The paper reports a 17% relative F1 improvement over GPT-4o on implicit "what" arguments and 14.7% on implicit "where/with". No absolute F1 value was found on Lacuna for iSRL-Qwen2-VL or GPT-4o on Implicit-VidSRL, so we will retrain it ourselves on the same silver training data, with the same visual inputs and a matched hyperparameter search budget, and report both our reproduction and the relative figures. We also test a hybrid in which our binding posterior is given to iSRL-Qwen2-VL as extra input, to see whether the two approaches are complementary. Beating this baseline means a statistically significant F1 gain at gaps of three or more steps with no loss at short gaps, plus lower expected calibration error overall.

Three ablations isolate the key ingredients. The deterministic entity memory, which is our model without stochastic states, isolates the value of uncertainty. A single global recurrent latent in the style of DRGD, with frames as input and a pointer over entities as output, isolates entity factoring. A frame-agnostic transition, in which every entity updates identically whether or not it was acted on, isolates the claim that structured roles should drive the state update. For each, beating it means a gain on the metric that ingredient is supposed to affect: calibration and ambiguous-slot coverage for stochasticity, long-gap F1 for factoring and frame-driven transitions.

For the secondary narrative-cloze check, the strongest reported numbers are 65.88% accuracy for contrastively fine-tuned BART-large (A Generative Approach for Script Event Prediction via Contrastive Fine-Tuning) and 65.19% for SSD-GM (Semantic and Sentiment Dual-Enhanced Generative Model for Script Event Prediction). There we would add our protagonist state as a feature to the BART scorer, not claim a standalone win.

Why not just…?

Why not just fine-tune a larger vision-language model on the tuples? This is the most serious threat. The iSRL-Qwen2-VL results show fine-tuning already helps a great deal, and more scale could close the long-gap errors. The evidence that it may not is the reported failure to track entity state across steps, together with the large gain from making structure explicit. The gap-length analysis settles the question: if the fine-tuned model's F1 does not decay with gap length, or our model does not flatten that decay, we will say so and treat the hybrid as the contribution.

Why not just use a deterministic entity memory? It may match our F1, and we expect it to be close. We keep the stochastic version only if it wins where uncertainty should matter: lower calibration error, higher posterior coverage of annotator-accepted fillers on ambiguous slots, and confidence that falls as gap length grows. If none of these hold, the conclusion is that frame-driven entity memory is enough, and we would report that.

Why not just calibrate a deterministic model with ensembles or temperature scaling? Post-hoc calibration can fix average overconfidence. It cannot make uncertainty grow specifically for entities that have gone unmentioned, and it cannot put mass on two distinct fillers for reasons tied to their states. We will compare against a five-member ensemble of the deterministic memory with temperature scaling, using matched total compute. Stochastic states must beat it on calibration within the long-gap and ambiguous buckets, not just overall.

Why not just prompt a frontier model? Zero-shot models do poorly on these structured tasks: GPT-3.5 reaches 28.36% and Llama 3.1 70B reaches 39.32% on narrative cloze (Semantic and Sentiment Dual-Enhanced Generative Model for Script Event Prediction), and GPT-4o trails fine-tuned iSRL-Qwen2-VL on implicit arguments. We include GPT-4o only as a reference point.

Decisive Checks

The main experiment runs on the manually verified Implicit-VidSRL test set of 231 videos and more than 2.5K frames. We report F1 for implicit "what" and "where/with" arguments, expected calibration error and negative log-likelihood of the gold filler, and F1 curves as a function of gap length. Three results would confirm the proposal: a flatter F1-versus-gap curve than iSRL-Qwen2-VL, better calibration than the calibrated deterministic ensemble, and higher coverage of annotator-accepted fillers on ambiguous slots.

A counterfactual probe tests whether the model tracks state rather than recency. We delete an intermediate step that transforms an entity, such as a transfer into the pan, and check whether the predicted "where/with" filler for a later step changes in the correct direction. We also swap two entities' mentions and check that the bindings follow.

Two outcomes would refute the proposal. If the deterministic memory matches the stochastic model on every calibration and ambiguity measure, the claim for latent stochasticity is refuted. If the frame-agnostic transition matches the full model, the claim that structured events should drive state updates is refuted.