Branch-Local Pointing: Generating Alternative Futures of a Scene Without Losing Track of Its Objects

When a person in a kitchen has just picked up a mug, several futures are plausible. They might fill it, put it back, or carry it to the sink. Each future involves the same mug, the same hand, and the same table. They differ in what happens to those objects afterward. Models that predict activity from video ignore this structure in one of two ways. Video scene-graph models parse a single observed timeline into objects and their relations, and they never imagine alternatives. Event-graph models from natural language processing do generate branching futures, but only over abstract text types such as "Attacker" or "Vehicle", not over objects someone has actually seen. We propose a generator that builds a tree of alternative futures one event at a time. Each event's participants are chosen by pointing at an object already observed in the video, or by explicitly declaring a new object. Each branch keeps its own copy of the scene's state, so the mug can be in the hand in one branch and back on the table in another.

The mechanism combines three existing ideas, and the combination is the contribution. The first is the copy gate from pointer-generator networks, a decoder that at each step either produces a symbol from its vocabulary or copies an item from its input. The second is step-by-step graph generation with action masking, where illegal moves are blocked based on the partially built graph. The third is a structured scene graph, in which nodes are tracked objects and edges are relations like holding or on. The key new element is that the pointer's targets are not fixed input tokens. They are object slots whose relational state is updated separately inside each branch. Preconditions such as "you must be holding something to put it down" are checked against that branch-local state before the next event is allowed. The test is whether this reduces two failures that sampling-based and language-model approaches suffer from in branching prediction: inventing objects that are not in the scene, and binding an event to an object in a state it cannot be in.

Core Claim

Our claim is that branching event prediction grounded in scenes fails mainly on argument binding, not on predicting event types. A generator that points at observed object tracks, carrying branch-specific relational state and checked by precondition masks, will match or beat strong alternatives at predicting event types. It will also substantially reduce hallucinated entities and precondition violations, and it will cover more of the valid alternative continuations at a fixed sampling budget. If the copy gate and branch-local state do not reduce binding errors relative to an otherwise identical model that emits entity IDs as ordinary vocabulary tokens, the claim is false.

Relation to Prior Work

The closest existing system is The Future is not One-dimensional. It generates event graphs autoregressively from news document clusters. It uses an edge-aware graph network and a copy mechanism that decides whether a new event's argument is a fresh entity or an earlier participant. It reports 58.15% Event Match F1 on LDC2020E25 and HITS@1 of 0.741 on the IED corpus. This paper supplies our method: we reuse its add-event-then-bind-arguments decoding and its copy-or-create decision. We differ in three ways.

  1. Its entities are type-level symbols extracted from text. Ours are tracked visual objects with relational state.
  2. It builds one merged schema graph. We keep a separate scene state per branch, so the pointer's target changes meaning depending on the branch.
  3. It has no physical preconditions to enforce.

Connecting the Dots also motivates the entity-anchored view, since events link through shared participants. Its path language model has no instance-level copying, and its schemas stay at coarse types. Those are exactly the limits we target.

On the scene side, Action Genome, STTran, and VirtualHome Action Genome provide the representation and the data. Their roles differ:

  • Action Genome treats actions as transitions in object relations, for example not contacting to holding. This motivates defining events by their effects on the scene graph.
  • STTran parses relations over a sliding window and lacks persistent tracking. It shows the identity problem exists even on one timeline.
  • VirtualHAG shows that simulator-generated labels are far more consistent than human annotation. This is why we can build branching ground truth in simulation.

All three assume one observed timeline and none of them generates.

CERES is the nearest scene-grounded event generator. It completes links over a five-frame temporal scene graph seeded from real driving logs and filters predictions with a traffic ontology. It does not generate structure from scratch, does not branch, and names longer multi-stage maneuvers as unsolved. Pointer Graph Networks learn node-to-node pointers that imitate classical data structures under supervised traces. That is evidence that learned pointers can track changing structure, but only on synthetic tasks.

The masking design comes from molecular and directed-acyclic-graph generation. CGVAE reached 100% chemical validity by masking against the partial graph. It also found that conditioning on the current partial graph, rather than the full construction history, reduced overfitting to arbitrary build orders. Zhang et al. and Deleu et al. mask cycle-creating edges in the same spirit.

Two papers constrain the design.

  • Choice-75. Choice-75 shows that large language models reach only 0.60 accuracy on branching script decisions, versus 0.74 for humans, and 0.22 on "either" cases. Branching is genuinely hard, but this is a choice task, not generation, so it does not by itself rule out language models as generators.
  • The Action Genome oracle result. Ground-truth scene graphs raise action recognition from 44.3% to 60.3% mAP. Any scene-grounded system will be limited by its object detector, so we must report results with both oracle and predicted graphs.

Research Program

The generator. It observes a video prefix, parsed into a tracked scene graph. It then grows a tree of future events. At each step it does four things:

  1. It chooses a parent event on the current frontier. Choosing an already-expanded parent creates a branch, so branching is itself a pointing decision over frontier nodes.
  2. It chooses an event type from a fixed vocabulary, such as grab, open, or put-on.
  3. It fills each argument role of that event type.
  4. It applies the event's effect to that branch's scene state.

Each branch $b$ maintains a set of object slots. Each slot $e$ carries a track embedding from the observed video and a relation state $s_{e}^{b}$, meaning its current edges in branch $b$. For a role $r$ of the new event, the argument distribution mixes copying and creating:

$$P(\text{arg}_r = e) = (1-g_r)\,\frac{m_{e}^{b,r}\,\exp(q_r^\top k_{e}^{b})}{\sum_{e'} m_{e'}^{b,r}\,\exp(q_r^\top k_{e'}^{b})}, \qquad P(\text{arg}_r = \text{new}) = g_r,$$

The symbols are defined as follows:

  • $g_r \in [0,1]$ is a learned gate for creating a new entity, whose type is then drawn from the object vocabulary.
  • $q_r$ is the decoder's query for role $r$.
  • $k_{e}^{b}$ is a key for slot $e$, computed by a graph network over branch $b$'s current scene graph.
  • $m_{e}^{b,r} \in \{0,1\}$ is a precondition mask. For example, the patient of put-on must be held by the agent in branch $b$.

Because the keys come from the branch's present state rather than from the original frame, the same mug can be a valid target in one branch and masked in another.

The effect step updates edges in the branch state after each event, such as adding holding and removing on-table. We study two versions:

  • Symbolic effects. The update is read from the simulator's action semantics. This is the ontology route that CERES advocates.
  • Learned effects. A relation-transition head trained on Action Genome-style state changes predicts the update. This is the data-driven route that The Future is not One-dimensional favors.

Running both versions directly tests the tension between learned flexibility and ontology-imposed validity. A second design axis tests CGVAE's finding. We compare conditioning the decoder only on the branch-local state against also feeding it the full serialized construction history.

Data. Real video gives one future per prefix, so we build branching ground truth in simulation. VirtualHAG is rendered in the VirtualHome simulator from activity programs. For each starting scene and goal, we execute several alternative programs that share a prefix and diverge afterward. We draw these variants from alternative script orderings and from decision points of the kind catalogued in Choice-75's proScript-derived goals. We record the resulting scene graphs. Every branch is then simulator-executable by construction, and every argument has an exact track identity. This yields three properties that real data cannot provide: a tree of valid futures, exact identity labels, and an executability oracle.

For real-world transfer, we train on single-future Action Genome sequences and evaluate prefix-conditioned next-event prediction with argument binding. We also run a small human-rated study of alternative continuations. Predicted scene graphs come from the one-stage dynamic scene-graph generator in OED, so we can measure how detector noise propagates into binding errors.

Baselines

Do-nothing reference. This is a frequency prior. It predicts the most common next event type given the last observed event, and binds each role to the nearest object of the right category in the last frame. It sets the floor for both type prediction and binding, and shows how much binding is solvable by proximity alone.

Strongest existing method. This is the generator from The Future is not One-dimensional, re-implemented over scene-graph symbols. Object categories plus track indices serve as entities, and its copy mechanism points over previously generated entity nodes. It is the only published system that autoregressively generates branching event graphs with copy-based participant identity, so it is the method to beat. Its reported numbers come from text corpora: 58.15% Event Match F1 on LDC2020E25 and HITS@1 of 0.741 on IED. No reported result found on Lacuna for any method on branching scene-grounded event generation in VirtualHAG or Action Genome. Every number on our task will therefore be produced by us under one protocol.

It receives the same observed scene graphs, the same training trees, the same decoder size, and the same hyperparameter search budget. Beating it means two things. First, a statistically significant reduction in precondition-violation and wrong-binding rates on held-out VirtualHome scenes. Second, Event Match F1 on branches that is no worse than its own, within confidence intervals.

Language-model baseline. This is a current instruction-tuned large language model. It receives the same serialized scene graph, with objects listed by ID and current relations. It is asked to output a tree of alternative continuations in a fixed grammar, with constrained decoding so that output always parses. It also gets the same number of few-shot examples we use for validation tuning. Choice-75 and CERES both show language models trailing dedicated models. However, those tests used older models and different tasks, so this baseline must be run rather than dismissed.

Linear-sampling baseline. This is our own generator with the branch-selection step removed. It produces one chain per sample, and we draw $k$ samples, with $k$ matched to the number of leaves our model produces. This isolates whether explicit branching with shared prefixes adds anything beyond sampling diversity.

Ablations isolating the key ingredient. Four ablations each remove one element of the method:

  1. Replace the copy gate and slot attention with entity IDs emitted as ordinary vocabulary tokens.
  2. Keep pointing but compute slot keys from the last observed frame rather than branch-local state.
  3. Remove the precondition mask, keeping it only as a training-time auxiliary loss.
  4. Swap symbolic for learned effects.

Why not just…?

Why not just prompt a large language model with the serialized scene? This is the most serious threat. Modern language models are good at producing plausible branches from a list of objects. The evidence against them is indirect. Choice-75 reports 0.60 accuracy on branching decisions, against a human 0.74. CERES reports that its graph model beat GPT-4 at scenario generation, with F1 of 0.706. Neither study measured identity binding across branches. Our experiment measures it directly. If the language model, with constrained decoding and the same serialized state, matches our wrong-binding and violation rates, the specialized architecture is unjustified. We would then reframe the contribution around the branching benchmark and the branch-local state representation, which could be fed to the language model as text.

Why not just sample a single-timeline forecaster many times? Sampling yields diverse futures. But it does not give the futures a shared prefix, and it does not let one branch's choice inform another. Most likely it will waste samples on near-duplicates. We test this at a matched leaf budget using branch coverage, meaning the fraction of ground-truth alternative continuations matched. If sampling covers as much at equal cost, explicit branch selection should be dropped and the paper reduces to pointing plus state tracking.

Why not just give each object an ID token and skip the copy gate? With a modern transformer, emitting an ID token is nearly pointing already. The real question is whether the gate and the branch-local keys matter. Ablations 1 and 2 answer this. We expect the gain to come mostly from the branch-local keys, which let identical-category objects in different states be told apart. The literature identifies diffuse attention over repeated entities as a known failure of pointers. If the gate alone accounts for nothing, we will say so.

Why not just filter outputs post hoc with an ontology, as CERES does? Post-hoc filtering discards invalid branches after generating them. That reduces coverage per unit of compute, and it cannot repair a branch whose early binding was wrong. Ablation 3 tests this, comparing violation rate and coverage at matched compute between in-loop masking and generate-then-filter.

Proposed Experiments and Decisive Checks

Main experiment. We train on VirtualHome branching trees and test on held-out scenes and held-out goals. Metrics are as follows:

  • Event Match F1 per branch, adapted from The Future is not One-dimensional by matching each generated branch to its best ground-truth branch.
  • Argument binding accuracy against exact track identities.
  • Hallucinated-entity rate, meaning new entities created when a valid observed object existed.
  • Precondition-violation rate, checked by executing each branch in the simulator.
  • Branch coverage at a fixed leaf budget.

Every result is reported under oracle scene graphs and under OED-predicted graphs. This separates our contribution from the detector bottleneck documented by Action Genome's 44.3% versus 60.3% oracle gap.

Transfer experiment. On Action Genome, we report next-event HITS@1 and binding accuracy from observed prefixes. We compare models trained only on Action Genome with models pre-trained on the VirtualHome trees. This mirrors VirtualHAG's report that simulator pre-training raised Action Genome PredCLS Recall@20 from 70.9% to 72.1%. A modest gain is the realistic expectation. We complement this with human ratings of alternative branches on a few hundred prefixes, scored for plausibility and for whether each branch refers to the right objects.

Decisive checks. Three outcomes would settle the claim:

  1. If ablation 2, which uses last-frame keys, matches the full model on binding accuracy in scenes with multiple same-category objects, branch-local state is not doing the work.
  2. If the language-model baseline matches the full model on violation rate under oracle graphs, the architecture is not needed.
  3. If the gap over The Future is not One-dimensional vanishes under predicted scene graphs, the method only helps with clean perception, and the paper must say so.

The result we expect, and that would justify the work, is this. Event-type F1 stays roughly flat across strong methods, while binding errors and violations drop sharply for the full model, and the drop is largest in branches that diverge from the observed state.