State-Keyed Retrieval for Inferring Past Actions from a Single Frame

A photograph of a kitchen with an open fridge, a milk carton on the counter and a person holding a glass shows no motion. It still records what just happened. Humans read these traces easily. Machines do not. On the task of listing the actions that led up to a single frame, people reach about 80.6 mean average precision (mAP), the best relational model reaches 35.75 mAP with perfect object boxes, and plain image encoders stay under 10 mAP (Abductive Action Inference). We propose to close part of this gap with a lookup rather than a larger model. From ordinary training videos, we build a memory that maps an end state, meaning the set of physical relations between a person and nearby objects such as "holding glass" or "fridge open, not touched", to the action sequences that actually produced that state. At test time, the relations detected in a single frame are used as the query into this memory. The retrieved histories, together with the detector's uncertainty about each relation, are passed into a language model, which then ranks or writes out the likely past actions.

The idea rests on two claims. The first is that knowledge retrieval for interaction scenes should be indexed by contact and spatial state, not by object names. The second is that the detector's relation probabilities should be passed to the language model as continuous projected tokens, not as text, because the main measured weakness of current methods is their collapse under noisy detections. Both claims can be tested directly on the Action Genome benchmark.

Core Claim

Inferring the past from one frame is hard for two reasons. Interaction histories are scarce, so a model cannot memorize which traces follow which actions. The frame also carries no motion, so the answer must come from how objects and people are arranged. Every annotated training video, however, supplies many free examples of "this relation state was preceded by these actions": each frame can be treated as an endpoint, and its preceding action labels are known.

We claim that a non-parametric memory built from these pairs and queried by relation state will outperform both entity-keyed retrieval and closed-book prediction, and that the advantage will grow as training data shrinks. We further claim that projecting soft relation probabilities into the language model will recover a large share of the 10–12 mAP drop that Abductive Action Inference reports when ground-truth boxes are replaced by Faster R-CNN detections. Serializing the same scene graph as text should not recover it, because text forces a hard decision about each relation. That second claim is the one most likely to fail, and the experiments are designed so that it can.

Relation to Prior Work

Some prior work motivates the gap. The abductive past-action work, Inferring Past Human Actions in Homes with Abductive Reasoning and Abductive Action Inference, defines the task on Action Genome. Its BiGED model uses a graph network with bilinear pooling over human–object–object triplets and static GloVe word vectors. It is closed-set and uses no retrieval. Its own ablations show that perception noise, not reasoning, is a large share of the error. Action Genome shows that actions are best described as transitions between relations: moving from oracle to predicted scene graphs changes Charades recognition from 60.3 to 44.3 mAP. That result assumes video input, and it supplies exactly the relation vocabulary we index on.

Other work supplies the method but constrains the design. EntRAG (EntRAG) binds image regions to named entities for knowledge retrieval and reaches 46.1 accuracy on E-VQA. We reuse its idea of scoring the local region and reranking, but we replace entity identity with relation state, since "cup" says nothing about whether it was just poured. DARK (Disentangled Action Recognition with Knowledge Bases) shows that external affordance priors mined from text lift open-world verb–noun recognition from 0.061 to 1.69 Macro Open AUC. Its verb branch needs a video backbone (I3D), so it cannot run on a single frame. Our memory plays a similar role, pruning implausible histories, without needing motion. Deep Pre-Alignment (Deep Pre-Alignment for VLMs) shows that a naive projector plus aggressive tuning drops a base model's MATH-500 score from 84.8 to 36.4. This constrains us: the language model stays frozen or is lightly adapted, and only a small projector over relation tokens is trained. Video-LLaVA (Video-LLaVA) shows the value of aligning inputs before a shared projector, but it projects raw frames rather than structured relations.

The last group is alternatives we reject for this setting. Hallucinating neighboring frames from video priors (Chen et al.) and AbductiveMLLM's pictorial "imaginer" (AbductiveMLLM) generate missing visual evidence instead of retrieving real histories. The latter reaches 57.04 CIDEr on VAR against a human 147.79, and it operates on video jump-cuts rather than contact traces. We use it as a comparison, not as a component.

Research Program

The system has four parts.

Perception. A detector produces person and object boxes. A relation head outputs, for each person–object pair $i$, a probability vector $p_i$ over Action Genome's 25 attention, spatial and contact relations. The vector is kept whole and is not thresholded into a single label.

Trace memory. It is built only from training-split videos. Every annotated frame $t$ contributes a key, its relation-state set, and a value, the ordered action labels active in the window before $t$. Retrieval scores a query frame $q$ against a memory frame $m$ by soft matching of relation distributions over aligned object categories:

$$ s(q,m)=\sum_{i \in q} \max_{j \in m,\; c_j=c_i} p_i^{\top} A\, p_j , $$

Here $c_i$ is object category and $A$ is a learned $25\times25$ relation-affinity matrix, initialized to the identity. With $A$ fixed to the identity and hard labels, the score reduces to exact state matching. This gives a clean ablation.

Projection. Each pair becomes one token $z_i = W[\,e(c_i);\, p_i\,]$, where $e(c_i)$ is the language model's own embedding of the category word and $W$ is a small trained projector. The top-$k$ retrieved histories enter as ordinary text. We compare a two-layer projector with a pre-aligned variant in the style of Deep Pre-Alignment for VLMs.

Output. For comparability with BiGED, the language model scores each of the 157 action labels by likelihood, which yields mAP and Recall@10. It can also generate free-form histories for qualitative analysis.

The data-scarcity question is addressed by training every system on 10%, 25%, 50% and 100% of Action Genome training videos. The memory shrinks proportionally at each fraction.

Baselines

The naive reference is a label-frequency prior conditioned on detected object categories. It shows how much of the task is solved just by knowing which objects are present, and it complements the reported sub-10 mAP of ResNet-101 and ViT features (Abductive Action Inference).

The strongest existing method is BiGED. It reports 35.75 mAP and 60.55% Recall@10 on "Abduct at T" with ground-truth boxes, and roughly 10–12 points less with Faster R-CNN detections (Abductive Action Inference). We rerun it with the same detector, the same training fractions, and a matched hyperparameter search of equal trial count. Beating it means exceeding 35.75 mAP with ground-truth boxes and, more importantly, losing less than half as much when switching to detected boxes.

The closed-book multimodal baseline is an open multimodal model of comparable size, fine-tuned on the same frames and labels with the same compute budget. We found no reported result on Lacuna for a multimodal LLM on Action Genome past-action inference, so this baseline is established here.

The key ablations share the same language model and budget. Each removes one ingredient:

  • no retrieval;
  • retrieval keyed by object names only (EntRAG-style);
  • retrieval keyed by relation state, with the scene graph written as text using argmax relations;
  • hard versus soft relation tokens;
  • $A$ fixed versus learned.

Why not just…?

Why not just fine-tune a strong multimodal model on the image? The reported sub-10 mAP of image encoders suggests that appearance alone does not carry the answer. A modern instruction-tuned model may do better than those encoders, though, and this is exactly the closed-book baseline. If it matches our system at full data, our claim narrows to the low-data regime. The scarcity curves test that directly.

Why not just write the scene graph as text and give it to the language model? This is the most serious threat. With ground-truth boxes, we expect text serialization plus state-keyed retrieval to match our system, because the relations are then certain. Our claim is limited to the detected-box setting, where soft tokens should keep information that argmax text discards. If the text variant closes the detection gap equally well, we drop the projection component and report retrieval alone as the contribution.

Why not just retrieve by object names, as in knowledge-based visual question answering? An object name fixes the objects but not what happened to them. An open fridge and a closed fridge share one name and point to different histories. The entity-keyed ablation measures this directly. If it performs as well, relation keys add nothing and Gap 2 is not real on this benchmark.

Why not just hallucinate the missing frames? Generating frames adds a heavy generator and can invent physically wrong transitions; AbductiveMLLM still trails humans by about 90 CIDEr on VAR (AbductiveMLLM). Retrieval returns histories that actually occurred. We will compare against a frame-hallucination variant fed into the same language model, on a held-out subset, at matched compute.

Proposed Experiments

All main results are on Action Genome's "Abduct at T" protocol. We report mAP and Recall@10 for ground-truth and detected boxes, at the four training fractions. Three checks decide the outcome.

The first check is the detection gap. We measure the drop from ground-truth to detected boxes for BiGED, for the text-serialized variant, and for the soft-token variant. The central prediction is that soft tokens cut the drop substantially, while text serialization does not.

The second check is the scarcity curve. At 10% of the data, state-keyed retrieval should beat both BiGED and the closed-book model by a wider margin than at 100%.

The third check is leakage and forgetting control. The memory is built strictly from training videos, and we verify that no test video's scenes are near-duplicated in it. We also compare the language model's MATH-500 score before and after projector training, against the forgetting reported by Deep Pre-Alignment for VLMs.

A secondary transfer test feeds the retrieved-history mechanism into VAR, measured against the reported 57.04 CIDEr of AbductiveMLLM. It is treated as exploratory, since VAR's missing events are narrative rather than physical.