Viewpoint-Indexed Memory for Streaming Camera-Controlled Video Generation

Video generators that run in real time, producing frames one chunk at a time while a user steers the camera, face a hard choice. To keep a character looking like itself, they keep attending to the first frames they made. Those first frames were seen from one particular camera angle. So the model also keeps pulling the scene back toward that angle. The result is that the camera stops responding, or the background tears when the user turns hard. If the model instead attends mostly to recent frames, the camera moves freely, but the character's face and clothes slowly drift. We propose to stop treating early frames as one undifferentiated anchor. Instead, the generator keeps a small memory of past frames, each tagged with the camera pose it was rendered from. At every step it uses that memory differently for two kinds of image regions. Regions on the moving subject retrieve stored frames whose viewpoint is closest to the current commanded view. Static scene regions attend only to recent history.

Our central bet is that the information needed to do this already sits inside a camera-controlled streaming generator. The commanded camera pose is known exactly, because the user supplies it. The 3D point tracks used for motion control separate pixels that move only because the camera moved from pixels that move on their own. That separation gives a per-token subject/scene split, computed in a single causal pass, with no segmentation model and no second rendering pass. If the bet holds, the method preserves identity when the camera returns to an earlier angle. It keeps the scene progressing and stays close to the base model's frame rate. No current streaming method does all three.

Core Claim

Two recent lines of work each solve half of this problem. Tether the Subject, Release the Scene (TetherMem) shows that uniform attention over cached history causes "memory-anchored scene under-progression." It fixes this by sending subject queries to historical subject memory and scene queries to recent history. Its masks, however, come from running SAM 2 on a first, unconditioned rollout and then regenerating. That two-pass scheme costs roughly 4× latency and is not causal. It also stores subject memory without regard to the viewpoint it came from.

Deep Forcing is training-free and runs at about 15.75 FPS on an H100. It keeps a deep attention sink of 40 to 60 percent of early context, realigns the temporal rotary position embeddings, and prunes historical tokens that recent attention ignores. It has no notion of geometry, so its sink pins the original viewpoint. 4DStreamCtrl reaches real-time camera and object control at 20.6 FPS, using 3D point tracks and a distilled 4-step causal student. Its frame-0 sink, though, is the very component behind its reported object-permanence failures during fast pans.

Our claim is concrete. A single-pass routing rule, with a subject memory retrieved by viewpoint, should beat both a frame-0 sink and recency-only attention on identity at revisited viewpoints, at matched camera-following accuracy. The masks come from separating camera-induced motion in the existing 3D tracks. The memory is indexed by the known commanded camera pose. The added latency should stay under about 10 percent.

What is new is the indexing of identity memory by viewpoint, which none of the cited systems does, together with getting the subject/scene split for free from the control signal. Subject/scene routing alone is not new; TetherMem already does it.

Research Program

The base system is a camera- and track-conditioned causal video diffusion student. A diffusion model is a generator that learns to turn noise into images by gradual denoising; a causal student is a fast few-step version that only sees past frames. We will use 4DStreamCtrl's released model if it is available. Otherwise we will use a camera-conditioned autoregressive backbone of the kind Deep Forcing is applied to, with Deep Forcing's sink and rotary realignment kept as the default. All of our changes operate at inference time, inside the attention layers, without retraining. A lightweight fine-tuning variant is a secondary arm.

The first component is the subject/scene split. At each chunk, every latent token has an associated 3D point track and the commanded camera transform $(R_t, T_t)$. We predict where each track would land if the world were static, by applying the camera motion alone. The residual between that prediction and the actual commanded track measures independent motion. Tokens whose residual exceeds a threshold, smoothed over the chunk, are labeled subject.

This is the same idea that motivates relative encodings in GoRela: a quantity defined relative to the camera is unaffected by the camera. Here we use it as a segmentation cue rather than a forecasting input. Because the camera pose is given by the user, we avoid the calibration dependence that limits AnyCamVLA. That system must know real camera parameters before it can re-render a test view into the training view.

The second component is the memory. Every few chunks we store the subject-labeled keys and values of one frame, tagged with the unit vector $d_k$ from the camera center to the subject's 3D centroid in world coordinates. When generating frame $t$, subject queries attend to the $m$ stored entries with the highest score:

$$s_k = d_k^\top d_t - \lambda \, \frac{t - t_k}{\tau}$$

Here $d_t$ is the current viewing direction toward the subject, $t_k$ is the time entry $k$ was written, $\lambda$ trades viewpoint match against age, and $\tau$ is the context length. The first term says that a frame seen from nearly the same angle is the best reference for what the subject should look like now. The weak age penalty breaks ties in favor of fresher appearance.

Retrieved keys get Deep Forcing's temporal rotary realignment so their positions do not cause flicker. Scene queries attend to a short recent window plus a small fixed sink for global color statistics. That sink is kept deliberately small, because Deep Forcing's evidence shows that removing sinks entirely brings back color drift.

The memory is bounded at a fixed number of entries, with eviction by viewpoint redundancy: when two entries see the subject from nearly the same angle, the older one is dropped. Memory and compute therefore stay constant over arbitrarily long streams.

We will also analyze the mechanism directly. The question is whether retrieval by viewpoint addresses the linear-interpolation effect that RefDrop identifies. RefDrop shows that sharing keys and values across frames interpolates latents and damps motion. Our hypothesis is that interpolating toward a reference seen from a matched angle damps identity variance without damping camera-induced change. That would show up as identity variance falling while measured camera following stays flat. We test this by sweeping reference strength under matched versus mismatched retrieval.

Baselines

The naive reference point is the backbone with a plain sliding window and no sink. It measures how much drift and stagnation occur with no memory policy at all.

The strongest existing method for this question is Deep Forcing applied to the same camera-conditioned backbone. It is training-free, it runs at streaming speed, and on VBench-Long it reports a Dynamic Degree of 57.19 against 31.35 for Rolling Forcing, at about 15.75 FPS on an H100. It therefore already handles the motion-freezing side of the trade-off. We will use its published sink fraction and pruning settings, and give it the same tuning budget we give our own threshold and $\lambda$: an equal number of configurations searched on a held-out set of camera trajectories.

To beat it, we must match or exceed its Dynamic Degree and its camera-following error, while clearly improving identity at revisited viewpoints. "Clearly" means outside the seed-to-seed variance we measure across three generation seeds per prompt. We must also stay within 10 percent of its frame rate.

The backbone's own frame-0 sink, as in 4DStreamCtrl, is the second baseline. That paper reports a DAVIS end-point error of 5.29, sustained coherence over 350 frames, and 20.6 FPS. Our camera following must not regress from it.

TetherMem is the reference for identity-versus-progression quality, without latency constraints. It reports a subject identity score of 0.600 and human expected preferences of 0.780 for overall quality and 0.769 for scene progression. Rolling Forcing scored slightly higher on identity in that paper but stagnated. We run TetherMem in its two-pass form as an upper-bound comparator. We also run a causal variant described in the next section.

Two ablations isolate the key ingredients. The first keeps our geometric routing but replaces viewpoint-indexed retrieval with a frame-0 subject sink. The second keeps viewpoint retrieval but applies it to all tokens with no routing. If the first ablation matches the full method, the memory indexing contributes nothing and the contribution reduces to cheap routing.

Why not just…?

Why not just run SAM 2 causally on the previous chunk and feed those masks to TetherMem-style routing? This is the most serious alternative, and we treat it as a real threat rather than a strawman. It removes the second rendering pass at the cost of one chunk of mask lag and a tracker's runtime. Two pieces of evidence suggest it may fall short under large viewpoint changes. In Segment Concept, SAM 2 drops to 58.2 J&F on the SeCVOS benchmark, which features extreme perspective changes, against 70.0 for concept-level tracking. And the lagged masks would be computed on frames the generator itself produced. We will run exactly this variant with matched retrieval disabled and then enabled. If it equals our geometric masks on identity and latency, we will report that the mask source does not matter. The claim then rests on viewpoint-indexed memory, which this alternative does not supply.

Why not just keep Deep Forcing's deep sink and add camera conditioning? This is our main baseline, so the experiments answer the question directly. The specific failure we expect comes from the sink itself. A sink holding 40 to 60 percent of early context encodes the initial viewpoint, so on orbit-and-return trajectories it should either resist the commanded turn or blend in features from the wrong side of the subject. If it does not fail this way on our trajectories, the proposal's premise is wrong. We would see that as equal revisit consistency with no loss in camera following.

Why not just train with multi-view augmentation or break inertia with noisy history, as Astra does? Astra injects noise into history so the model follows actions. It reports 0.641 instruction following on CityWalker against 0.619 for YUME. The file notes, though, that this noise weakens the identity anchor during sustained turns. AnyCamVLA reports that viewpoint-augmented fine-tuning behaves like a zero-sum game, gaining on unseen views while losing on original ones. We include Astra-style noise augmentation on the history as a training-free baseline, measured on the same trajectories. We do not pursue augmentation-based retraining as the main line, because it costs far more compute than an inference-time memory policy.

Proposed Experiments

No existing benchmark directly measures identity when a camera returns to a previously seen angle. We therefore build a trajectory suite on top of VBench-Long prompts, the MovieGen 128-prompt set used by Deep Forcing. Each prompt is paired with three camera programs: a 180-degree orbit that returns to the start, a pan away from and back to the subject, and a slow continuous rotation of the kind Segment Concept's color-histogram trigger is known to miss. Each stream runs at least 350 frames, the horizon 4DStreamCtrl reports.

The headline metric is revisit consistency. It is the DINO feature similarity, using the same backbone as VBench's subject consistency, between subject crops generated at matched commanded viewpoints separated by long intervals. Crops are taken from the geometric masks and checked with a held-out segmenter to avoid circularity. Alongside it we report several standard measures. From VBench++ we take dynamic degree, background consistency, and imaging quality. From 4DStreamCtrl's DAVIS protocol we take end-point error. We add camera rotation and translation drift in the style of Astra-Bench, and measured FPS on one GPU. A small human preference study follows TetherMem's protocol for overall quality and scene progression.

Three results would settle the question. First, on orbit-and-return, the full method should beat Deep Forcing and the frame-0 sink on revisit consistency while matching their end-point error and dynamic degree. Second, the first ablation, frame-0 subject sink with routing, should lose most of that gain, showing that viewpoint indexing is what matters. Third, on stationary-camera prompts the method should match Deep Forcing. That would show we have not traded away the regime where a single sink is already sufficient.

A secondary test checks robustness of the geometric masks. We perturb track depth to simulate the monocular scale errors 4DStreamCtrl reports and measure how quickly routing accuracy and revisit consistency degrade. A final test probes multi-subject interaction, which TetherMem cannot handle. There we cluster the motion residuals into separate subjects, keep one memory per cluster, and check whether attributes still bleed across subjects when they cross.