Keeping World-Model Rollouts Geometrically Honest with Sparse In-Sensor Tracks

A video world model is a generator that predicts future frames of a scene given the past. When such a model runs on a robot or drone, it drifts. Walls bend, objects slide off the surfaces they rest on, and a room seen again after a loop no longer matches the room seen first. The usual fix is to keep showing the model real camera frames. Streaming full frames off the image sensor, however, is the most power-hungry step in an embedded vision pipeline, the "data tax" described in A processing-in-pixel-in-memory paradigm for resource-constrained TinyML applications. Meanwhile, pixel processor arrays can already track point features inside the sensor at thousands of frames per second. These are chips that put a tiny processor behind every pixel. They send out only coordinates, cutting off-chip bandwidth by more than 1000x (Descriptor-In-Pixel : Point-Feature Tracking For Pixel Processor Arrays).

We propose to connect the two. The sensor emits only sparse 2D feature tracks. The world model predicts both images and per-pixel 3D coordinates. At every step, the model's predicted 3D structure is forced to reproject onto the tracks the sensor actually observed. The bet is that geometric drift has few degrees of freedom: it lives in camera pose and scene structure, not in texture. If that is right, a few hundred tracked points per frame, at roughly a hundredth of the bits of a video stream, should anchor geometry about as well as real frames do. The claim fails if a compressed video thumbnail at the same bit rate anchors geometry just as well. The central experiment is built to test exactly that.

Core Claim

At a fixed off-sensor bit budget, sparse feature tracks give more geometric correction per bit to a world model than any pixel-based stream of the same size. This advantage should hold only if the model predicts explicit 3D structure that the tracks can be checked against. Concretely, suppose the model predicts a pointmap $\hat{X}_{t_0}$ (a 3D point for every pixel of a reference frame) and camera poses $\hat{T}_t$. The sensor reports track positions $u_i^t$ for points first seen at pixel $u_i$. We measure disagreement with the reprojection residual

$$ R = \sum_{i,t} \rho\big(\lVert \pi(\hat{T}_t\, \hat{X}_{t_0}(u_i)) - u_i^t \rVert\big), $$

where $\pi$ is the camera projection and $\rho$ is a robust penalty that downweights outliers. The residual needs only 2D observations, so we never have to lift the chip's 2D tracks into metric 3D on the sensor. That lifting is the barrier the research file identifies between in-sensor tracking and the 3D point regularization of Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach. The world model supplies the 3D structure, and the sensor only checks it.

Relation to Prior Work

The research file finds no work joining these directions. The closest pieces play different roles here.

Some prior work motivates the proposal by showing that 2D generators lack geometry and that explicit constraints help. Geometry Forcing shows that video diffusion features barely support linear probes for depth. It aligns them with a frozen VGGT geometry model, which cuts 256-frame FVD on RealEstate10K from 364 to 243, but it still drifts because it never sees new observations. CamI2V and CamCo restrict cross-frame attention to epipolar lines, the lines along which a point can appear in another view given the camera motion. CamCo reaches 3.8% COLMAP reconstruction error against 64.3% for base Stable Video Diffusion. Both methods take camera poses as offline ground truth. We instead take evidence about pose and structure from a live, bandwidth-limited sensor.

Other prior work supplies the method. Geometry-aware 4D Video Generation for Robot Manipulation jointly generates RGB and pointmaps, and we reuse that output head so the residual $R$ can be computed. The point-regularization paper shows that 3D point trajectories curb morphing, raising VBench subject consistency from 0.837 to 0.960. It also names dependence on heavy offline trackers as its weakness. We replace that tracker with an on-sensor one and move the supervision from training time to inference time.

The in-sensor papers supply the measurement stream and also constrain it. Descriptor-In-Pixel tracks at 3012 FPS, taking 321 µs per frame at about 1 W, but it keeps only about 20% of features after 15 seconds and its 8-bit descriptors are not viewpoint-invariant. Focal Plane Visual Feature Generation and Matching on a Pixel Processor Array matches descriptors of up to 256 bits on chip, but it relies on the FAST corner detector, which is fragile. Visual Odometry for Pixel Processor Arrays estimates pose on chip at 1–1.5 kHz, but only as a 2D affine approximation. These limits mean the tracks will be short-lived, noisy, and outlier-prone. That is why $R$ uses a robust penalty and a changing set of tracks rather than assuming long clean trajectories.

Research Program

We build an autoregressive video diffusion model with Geometry-Forcing-style alignment to VGGT and a pointmap head in the style of the 4D robot generator. It is conditioned on camera-free past context. Track information enters in two ways, which we ablate separately.

The first route is conditioning. Active tracks are rasterized into a sparse displacement map, with zeros where no track exists, and fed as extra input channels during training.

The second route is guidance at sampling time. At each denoising step, we form the model's current estimate of the clean pointmap and poses, compute $R$, and nudge the sample down the gradient of $R$, in the manner of reconstruction guidance. The on-chip trackers run at kilohertz rates while the generator runs at a few frames per second. Between generator steps, tracks are integrated into long-baseline correspondences. The speed mismatch that the research file frames as a barrier therefore becomes an advantage, because longer baselines constrain depth better.

Most experiments cannot run on SCAMP hardware, so we build a sensor emulator. It converts video to 256×256 grayscale, detects features the way the chip does, stores 8-bit BRIEF-like descriptors, searches within 9×9 patches as Descriptor-In-Pixel does, quantizes coordinates, and drops tracks to match the published retention behavior. On 30 fps RealEstate10K footage, the emulator is deliberately handicapped relative to a real 3 kHz chip, because larger inter-frame motion makes tracking harder. Positive results are therefore conservative. We will calibrate the emulator against real SCAMP-7 recordings if the hardware groups share them, comparing track retention and error statistics.

Bit rates are computed exactly. Roughly 200 tracks at 30 Hz, each carrying an ID and two 8-bit coordinates, come to about 150 kbit/s. Raw 256×256 8-bit grayscale video at 30 Hz is about 15.7 Mbit/s. This is our own arithmetic, not a reported figure. We report bits, not joules. The energy argument rests on the published chip power figures, not on new measurements.

Baselines

The naive reference point is open-loop rollout. The same world model is given the initial context and then receives no further observations. This measures how much drift there is to correct.

The strongest existing method for geometric consistency without observations is Geometry Forcing. We run it as the open-loop model, since our base model includes its alignment losses, and we train a faithful copy on RealEstate10K at matched compute. Its reported 256-frame FVD of 243 serves as a sanity check that our reimplementation is sound. It is not a target, because our closed-loop setting receives information that it does not.

The competitor that actually matters is rate-matched frame re-conditioning. Here the same model is periodically re-conditioned on real frames that have been H.264-encoded and downsampled so that the bits per second match the track stream exactly. We sweep resolution against refresh rate and pick the best setting per budget on validation data, with the same tuning budget as our guidance-strength sweep. Beating it means lower pointmap and pose error at equal bits across most of the rate range. Matching it would be a negative result.

A pose-only baseline feeds the on-chip visual-odometry output, only six numbers per step, through CamI2V-style epipolar attention with Plücker ray embeddings, which encode each pixel's viewing ray. This baseline isolates whether tracks help beyond what camera motion alone provides.

The key ablation removes the pointmap head. Tracks are still given as conditioning, but no 3D prediction exists to reproject against. If this matches the full system, the explicit-geometry part of the claim is wrong.

Why not just…?

Why not just send a compressed low-resolution video stream instead? This is the real threat. A 128-pixel H.264 stream at 150 kbit/s preserves coarse layout, and a strong generator might recover geometry from it. We do not argue this away. The rate-matched re-conditioning baseline tests it directly across a sweep of bit budgets, and we will report the crossover point if one exists. Our prediction is that thumbnails lose the fine, precise correspondences that pin down depth and pose. There is supporting evidence: codecs that reach the >1000x reduction reported for Descriptor-In-Pixel destroy high-frequency detail. However, the file contains no reported number for geometric accuracy under codecs, so the experiment decides.

Why not just send the on-chip camera pose and use a camera-controlled generator? CamCo and CamI2V show that pose conditioning fixes camera motion, which accounts for much of the drift. Pose says nothing about scene structure, though. Revisit error, which checks whether a revisited room matches, and object-level warping should persist under pose-only conditioning, and the pose-only baseline measures exactly this. If pose-only closes most of the gap, the proposal shrinks to "in-sensor visual odometry for camera-controlled world models," which is a smaller but still useful result.

Why not just scale the world model and let it learn consistency? Geometry Forcing's probing results and CamCo's 64.3% reconstruction error for base Stable Video Diffusion argue that scale alone does not produce usable geometry. More fundamentally, no amount of scale lets an open-loop model know where the real camera went. Only observations can supply that. The open-loop baseline quantifies how large this gap is.

Proposed Experiments

The main experiment runs on RealEstate10K. The model conditions on 16 frames and rolls out 256, receiving emulated tracks or the rate-matched alternatives throughout. Because the observed video is real, ground truth exists at every step. We measure pointmap error against VGGT pseudo-labels on held-out true frames, camera pose error against the dataset poses, and reprojection and revisit error as defined by Geometry Forcing (the file reports no exact values for these). We also report COLMAP reconstruction error of the generated frames following CamCo, and LPIPS and FVD against the true continuation. Every metric is plotted against bits per second.

The decisive check is whether the track curve lies below the thumbnail curve for geometric error at budgets under a few hundred kbit/s, while the thumbnail curve wins on appearance metrics. We predict exactly this split, because tracks carry geometry and thumbnails carry texture. If thumbnails win on geometry as well, the core claim is refuted.

A second check targets the static-world assumption shared by epipolar and VGGT methods. On the CamCo dynamic WebVid subset, we test whether the robust penalty rejects tracks on independently moving objects without corrupting the background, and we report per-region errors.

A third check runs in the Drake tabletop simulator used by the 4D robot generator, where emulated tracks come from rendered views. We ask whether track-guided pointmaps improve the 64% policy success reported there when the generator is re-anchored mid-execution instead of run open-loop. We also measure how much the added guidance increases the roughly 30-second generation time.

Finally, a small hardware study with real SCAMP-7 track logs checks that the gains measured in emulation survive real analog noise and track loss.