Keeping Identities Straight When Planned Subjects Cross Paths in Video Generation

Text-to-video models often get multi-subject prompts wrong. Subjects go missing, end up in the wrong place, or blend into each other. A common fix splits the work in two. A large language model (LLM) first writes a "layout plan", which here means a bounding box for each subject in each frame. The video generator is then steered so that each subject's word attends to its own box. This steering acts on cross-attention, the mechanism that ties each prompt word to image regions. The approach is effective. It breaks down, however, in the situation multi-object tracking researchers know best: two subjects whose paths cross. When the boxes overlap, nothing in the steering says which subject owns the shared pixels. Once the subjects separate again, the "red car" can come out of the crossing in the "blue car's" path. Tracking calls this an identity swap. In generation it shows up as swapped trajectories or as one subject bleeding into the other.

This note proposes treating the generator's attention maps the way a tracker treats noisy detections. The LLM plan is extended with a front-to-back ordering of subjects in every frame. Guidance pulls each subject only toward the part of its box that is not hidden behind a nearer subject. A lightweight data-association monitor, run across denoising steps and frames, detects when a subject's attention has latched onto another subject's planned track and re-applies guidance only where that happened. The goal is to measure, for the first time, how identity persistence degrades as the number of subjects and crossings grows, and to show that occlusion-aware, association-monitored guidance keeps it flat where current methods collapse.

Core Claim

The claim is that identity swaps in planned video come from the guidance step, not the planning step. The current energy functions are occlusion-blind: they reward attention mass anywhere inside a box. So once boxes overlap they cannot tell "A in front of B" from "A and B merged". The proposed fix has two parts. The first is a visibility-aware target region, each subject's box minus the boxes of subjects planned to be in front of it. The second is a check, borrowed from data association, of which planned track each subject's attention actually follows. Together these should remove most swaps without retraining the generator. A second claim is that this is a scaling problem. The gap between occlusion-blind and occlusion-aware guidance should widen with the number of subjects $N$ and the number of planned crossings $C$, because overlapping box pairs grow roughly as $N^2$ in crowded scenes.

Relation to Prior Work

Two training-free systems supply the base method. LLM-grounded Video Diffusion Models (LVD) has GPT-4 write per-frame box sequences. It then steers an off-the-shelf video model with cross-attention energy terms and center-of-mass tracking, and reports 98% layout-generation success against 14% for retrieval. Comp-Attn adds two pieces: re-injection of embeddings for subjects that other prompt words have suppressed, and soft, IoU-guided modulation of attention toward LLM boxes. It reports gains of +15.7% on Wan2.1-14B and +29.8% on CogVideoX-5B on T2V-CompBench at a 5.1% latency cost. Both control who appears and where. Neither represents depth order. Neither checks whether a subject's attention stayed on its own track through an overlap. The research file argues they have not been shown to hold beyond three or four subjects with intersecting paths, but it gives no measurement of this, and there is no reported result found on Lacuna for identity persistence under crossing trajectories. We keep their planning-plus-energy structure and change the target region and the control loop.

The tracking literature motivates the gap rather than supplying a method. The analysis of SimTrack notes that putting several target templates into one attention sequence causes target–distractor entanglement. Its 70.5% AUC on LaSOT is a single-target result. The detection-to-tracking work in Bridging the Gap Between Detection and Tracking needs separate modulation for each target identity. The shared lesson is that attention does not keep identities apart on its own; something explicit must do it. Our monitor is that explicit piece, kept deliberately cheap: per-frame bipartite matching, not combinatorial multi-hypothesis search.

Lifting the plan into 3D is the other relevant line. 3D Space as a Scratchpad arranges meshes in 3D, renders depth maps for images, and reports a 32% text-alignment gain on GenAI-Bench over 2D layouts. That result constrains our design: it is evidence that 2D boxes alone lose occlusion information. We take the lightest version of the same insight, an ordinal depth order emitted by the LLM. We do not build meshes, which would be costly and rigid for articulated subjects. Finally, Self-correcting LLM-controlled Diffusion Models corrects layouts after generation. Our monitor corrects inside the denoising loop, and the difference in compute cost is tested directly.

Research Program

The plan format extends LVD's per-frame boxes. Each frame also carries an integer depth rank per subject, and each subject a short appearance tag. The LLM is prompted to keep ranks consistent with motion; for example, the overtaking car is nearer. A small script checks that ranks are a valid total order whenever boxes overlap and asks the LLM to repair any violation.

The guidance energy for subject $k$ at frame $f$ is

$$E_{k,f} = 1 - \frac{\sum_{p \in V_{k,f}} A_{k,f}(p)}{\sum_{p} A_{k,f}(p)}, \qquad V_{k,f} = B_{k,f} \setminus \bigcup_{j:\, r_{j,f} < r_{k,f}} B_{j,f}.$$

Here $A_{k,f}(p)$ is the cross-attention from subject $k$'s tokens to pixel $p$. $B_{k,f}$ is its planned box. $r_{j,f}$ is subject $j$'s depth rank, with smaller meaning nearer. $V_{k,f}$ is therefore the visible part of $k$'s box. Fully hidden frames, where $V_{k,f}$ is empty, get no presence term. Instead they get a weak penalty on attention mass falling inside the front subject's visible region. This keeps the hidden subject from "borrowing" the occluder's pixels.

The monitor runs at a few denoising checkpoints. For each frame it summarizes every subject's attention as a centroid plus spread. It builds a cost matrix between subjects and planned boxes, adds a temporal term that penalizes jumps from the previous frame's assignment, and solves a bipartite matching. A swap is flagged when subject $k$ is matched to some other subject's track for several consecutive post-crossing frames. Guidance strength is then raised for those subjects and frames, and only for them. This targeted step is what separates the method from simply turning up global guidance. It also keeps added latency close to Comp-Attn's.

Baselines

The naive reference point is the unguided base model, CogVideoX-5B and Wan2.1-14B, given the same prompt and seed. These are the two backbones on which Comp-Attn reports its numbers.

The strongest existing method is Comp-Attn, with reported T2V-CompBench gains of +15.7% (Wan2.1-14B) and +29.8% (CogVideoX-5B). It is the most recent method that combines LLM boxes, attention modulation and a fix for suppressed subjects on these exact backbones. LVD serves as a second planned-guidance baseline, using center-of-mass energy.

For fairness, all guided methods receive the identical LLM plan for each prompt, with depth ranks removed for methods that cannot use them. This separates execution quality from planning quality. All methods also get the same seeds, sampling steps and a matched hyperparameter sweep over guidance strength, chosen on a held-out split.

A compute-matched SLD-style baseline regenerates up to $m$ times whenever a crop-level check flags a swap. Here $m$ is set so that its wall-clock time equals ours.

Two ablations isolate the ingredients. The first uses visibility regions only, with no monitor. The second uses the monitor with occlusion-blind boxes.

Beating the baselines means two things on the crossing stress set, defined below. Identity persistence must improve over Comp-Attn by a margin outside seed-to-seed variance at $N \ge 4$. And there must be no loss on T2V-CompBench or VBench quality dimensions relative to Comp-Attn.

Why not just…?

Why not just add depth order and hard masks, with no association monitor? This is the most serious threat, and it may be enough. If visibility-aware regions alone remove most swaps, the monitor is unnecessary and the contribution shrinks to "plan depth ranks". The visibility-only ablation tests this directly. We expect the monitor to matter in long or full occlusions, where the hidden subject has no target region and can reappear attached to the wrong track. If it does not matter there, we will report the simpler method as the result.

Why not just regenerate on failure, as in self-correcting pipelines? Self-correcting LLM-controlled Diffusion Models shows post-hoc correction works for images. For video, however, swap probability rises with crossings, so rejection sampling should need more retries per prompt as $C$ grows. The compute-matched regeneration baseline, measured as a function of $C$, settles whether the in-loop monitor is cheaper for the same persistence.

Why not just use a bigger backbone? Comp-Attn's gains on the 14B Wan2.1 model show that scale has not removed subject omission or misplacement. Whether scale fixes identity swaps specifically is untested. We therefore report every method on both backbone sizes and check whether the gap narrows at 14B.

Why not render a full 3D scratchpad? 3D Space as a Scratchpad relies on rigid meshes for each subject, which fit poorly with moving, articulated subjects in video and add heavy per-prompt cost. Ordinal depth is the minimal piece of 3D information the occlusion problem needs. If it falls short, depth-map conditioning becomes the next step.

Proposed Experiments

Broad compositional ability is measured on T2V-CompBench and the 500-prompt LVD complex benchmark. VBench checks that video quality does not drop.

The decisive test is a new crossing stress set, generated programmatically in the style of LVD. It varies the number of subjects $N$ from 2 to 8 and the number of planned path crossings $C$ from 0 to 6. Subjects are visually distinct, differing in category and color, so that identity can be checked.

The primary metric is identity persistence. For each subject we crop its planned box in every frame where it should be visible. We then ask a disentangled VQA scorer, in the style of T2I-CompBench, whether the crop shows that subject's appearance tag. A track counts as persistent if the answer stays positive after every crossing it takes part in. We validate this metric against human annotation on a subset before relying on it. Secondary metrics are swap rate per crossing, subject presence, VBench quality and added latency.

The headline plot shows persistence against $N$ and $C$ for every method. The prediction is that Comp-Attn and LVD decline steeply with $C$, while the full method stays roughly flat. The visibility-only ablation should sit in between, and the gap between visibility-only and the full method should widen in full-occlusion prompts.

A second check plants plans with deliberately wrong depth ranks. Persistence should drop toward the occlusion-blind baseline under these plans, which confirms the depth signal is actually being used. Third, latency is reported against Comp-Attn's 5.1% overhead, and the compute-matched regeneration baseline is reported at every $C$.

The idea is falsified if Comp-Attn's persistence is already flat in $C$ on the stress set, or if visibility-only matches the full method everywhere. Either outcome is useful, and we will report it as such.