Persistence-Guided Foveation for Video Scene Graphs

Video scene graph models describe a video as a set of tracked objects and the relationships between them over time, for example "person holding cup" becoming "person drinking from cup". They break in two places that matter for robots and surveillance cameras. Small objects get blurred away when the network shrinks the image to keep computation affordable. Real cameras also add noise that simulators never produce. The usual fix for the first problem makes the second worse. Methods that keep fine detail for tiny objects decide where to spend high resolution from a single frame's appearance, and on a noisy real camera a speck of sensor noise can look just as "important" as a distant cup. We propose to let time make that decision instead. A real object, however small, stays put or moves smoothly from frame to frame. Most sensor noise does not. The scene graph the model has already built tracks where its objects are and how they move, so it can say in advance where the next frame deserves full resolution.

The concrete proposal is a video scene graph model with a fixed budget of high-resolution image patches per frame. The budget goes to regions whose importance persists along the motion predicted by the model's own object tracks, and the rest of the frame is seen only coarsely. Because the budget is fixed, attention cost no longer grows quadratically with frame resolution or video length. Because allocation requires persistence over time, uncorrelated noise does not win high-resolution tokens. This yields a falsifiable prediction: the method should help under temporally uncorrelated noise, and it should fail in a predictable way under noise that is itself persistent and moves with the scene.

Core Claim

Every existing method that reallocates resolution toward small things scores importance per image. SG-Former estimates a significance map from attention and gives dense tokens to salient foreground while merging the background. TCFormer clusters tokens by similarity so hands and faces keep fine tokens. LIP learns a pooling weight map that stops small objects from being swamped by background. All three were developed and evaluated on clean static images. The research file identifies the risk directly: amplifying weak local signals risks amplifying sensor artifacts. Our claim is that in video, importance should be filtered along object motion before it controls resolution, and that the scene graph's tracked entities are the right source of that motion.

Let $a_t(p)$ be the single-frame importance of patch location $p$ at frame $t$, computed as in SG-Former. Let $W_t$ map locations in frame $t$ back to frame $t-1$ using the velocities of the tracked entity queries, falling back to identity in background regions. We define a persistence score

$$s_t(p) = \alpha \, s_{t-1}(W_t(p)) + (1-\alpha)\, a_t(p),$$

where $\alpha \in [0,1)$ sets how much history counts. The top-$K$ locations by $s_t$ receive full-resolution patch tokens. Everything else is represented by coarse stride-32 tokens. Suppose the noise added to $a_t$ is independent across frames with variance $\sigma^2$, and the warp is correct. Then this exponential average shrinks the noise variance to $\sigma^2(1-\alpha)/(1+\alpha)$, which is the standard steady-state variance of a first-order recursive filter, while leaving a persistent object's importance intact. That gap is the whole mechanism.

Two consequences follow, and the proposal is built to test both. First, newly appearing objects are delayed by roughly $1/(1-\alpha)$ frames. We therefore reserve a small exploration share of the budget for high single-frame importance, which must be confirmed over two or three frames before it is promoted. Second, noise that is correlated over time and also moves consistently with the warp, such as motion blur that follows object motion, is not filtered. Noise that is fixed in image coordinates, such as hot pixels or lens flare under a moving camera, is not consistent with object motion and should be filtered. The noise suite below separates these cases on purpose.

Research Program

The base architecture follows TCDSG, which uses a DETR-style detector whose object queries are carried across frames by a recurrent feedback loop. Training matches predicted and true tracklets over whole sequences, and this raised temporal recall on Action Genome well above post-hoc tracking. We keep its entity queries and its sequence-level matching. We change what those queries attend to.

In each frame, the entity queries cross-attend to a token set that has two parts: a coarse tokenization of the whole frame and $K$ fine patches chosen by the persistence score. The cost per frame is about $M(N_c + K)$, where $M$ is the number of entity queries and $N_c$ the number of coarse tokens, instead of growing with the full patch count. Relations across time are modeled only among entity queries. This keeps the temporal stage independent of pixel count. It also replaces the fixed sliding window of STTran with recurrent query memory, which carries context past any window.

The fine-patch embeddings are pooled with LIP-style learned weights, so detail inside a selected patch is not averaged away. Training uses temporally correlated noise augmentation on clean or rendered video. We separately test whether a gradient-reversal domain classifier on the entity queries, the device Sim2SG used to make static scene graph features appearance-invariant, adds anything once persistence filtering is in place.

The analysis side matters as much as the model. We will build a controlled sensor-noise suite applied to real video in three families. The first is temporally independent noise: shot and read noise, and random pixel or patch dropout. The second is fixed-in-image artifacts: hot pixels and static flare under camera motion. The third is motion-coupled degradation: blur along object motion and exposure flicker. We will apply each family at calibrated severities. We will also create a resolution-degradation axis by downsampling frames so that objects fall into size bins below 16, 16 to 32, and above 32 pixels on a side.

Baselines

The naive reference point is STTran with its frozen Faster R-CNN proposals at standard input resolution, trained on clean data. STTran reports a 1.9% PredCLS gain over GPS-Net on Action Genome; the file gives no absolute number. We include it because its frozen detector is exactly the bottleneck at issue: an object the detector misses cannot be recovered downstream.

The strongest existing method is TCDSG. It reports Action Genome temporal Recall@50 rising from 18.6% to 39.1%, at about 20 frames per second. It is the strongest competitor because it already solves the flicker and identity problems that our mechanism depends on. We will retrain it on the same data with the same noise augmentation, the same backbone, and the same hyperparameter search budget as our model. We will report it at its native compute and at our compute.

The key ablation is TCDSG plus SG-Former's per-frame significance allocation with the same $K$ fine-token budget. This is our model with $\alpha = 0$. It isolates whether temporal persistence, rather than adaptive resolution as such, is responsible for any gain. A second ablation replaces track-based warping with identity warping. This tests whether the scene graph's motion estimate matters or whether plain temporal averaging suffices.

We also include three stronger engineering baselines. The first is brute-force resolution: TCDSG with doubled input resolution. The second is a restoration front end, in which VmambaIR, a linear-cost state space restoration network, denoises frames before TCDSG. The third is full-resolution linear attention using QT-ViT's second-order Taylor kernel in place of token selection.

Beating the field means three things. On the small-object bins under temporally independent noise, our model must exceed TCDSG and the $\alpha=0$ ablation in temporal Recall@50 at matched FLOPs. It must stay within noise of the 39.1% tR@50 figure on clean Action Genome. And it must recover at least half of the clean-to-noisy drop that TCDSG suffers on small objects.

Why not just…?

Why not just feed higher-resolution frames to TCDSG? This is a real threat to the accuracy claim. Its cost grows with pixel count in the detector and with token count in attention, so for long or high-resolution video it does not scale. More importantly, higher resolution also feeds the model more noise at higher fidelity. We run it with no compute cap as an upper reference. If it matches us on noisy small objects at comparable cost, the persistence mechanism is not needed, and we will say so.

Why not just denoise first with a restoration network? Restoration networks are trained to remove high-frequency content that does not fit image statistics. A distant 8-pixel object is exactly such content, so a denoiser may erase it. VmambaIR's reported gains are in peak signal-to-noise ratio, which does not measure whether tiny objects survive. The VmambaIR front-end baseline settles this directly. If it closes the gap on small objects, the honest conclusion is that a good denoiser plus TCDSG is enough.

Why not just use per-frame adaptive tokens with heavy noise augmentation? Sim2SG's KITTI results show plain domain randomization trailing structured adaptation: 0.234 versus 0.316 mAP, and 0.070 versus 0.139 Recall@50. Those results are for static images, though, and do not test adaptive resolution. The $\alpha=0$ ablation with identical augmentation is exactly this alternative. If it matches our model, temporal persistence contributes nothing.

Why not just apply linear attention to every full-resolution token? This avoids selecting tokens at all. However, QT-ViT notes that truncated-kernel attention struggles to focus sharply on localized multi-object boundaries, which is where relationship predicates live. The full-resolution QT-ViT baseline tests this at matched FLOPs.

Why not just link per-frame scene graphs with an off-the-shelf tracker? TCDSG's own comparison shows post-hoc linking produces fragmented tracklets. Our method also needs the tracks inside the model to drive allocation, so a post-hoc tracker cannot provide them. We include it only as a secondary reference.

Proposed Experiments

The primary testbed is Action Genome. It has 10,000 videos, 35 object categories and 25 relationship categories, and is evaluated with Recall@K and temporal Recall@K on SGDET, the detection setting where object detection is not given. We report results under the noise suite and the downsampling bins.

The second testbed is PVSG, whose long EPIC-KITCHENS and Ego4D clips contain many small handheld objects. It is very hard: the best model reaches Recall@20 of 3.88% at 0.5 IoU in Panoptic Video Scene Graph Generation. Small absolute gains will be reported with confidence intervals across seeds. MEVA surveillance video provides naturally tiny people at range. TCDSG's MEVA result is aggregated with the other datasets, and no reported result was found on Lacuna for MEVA in isolation, so we will establish that baseline ourselves.

For the sim-to-real sensor gap, we train on synthetic PSG4D-GTA from 4D Panoptic Scene Graph Generation and evaluate on real egocentric PSG4D-HOI. Evaluation is restricted to a manually mapped set of shared object and relation categories, using RGB-D input. Content differs between the two sets as well as appearance, so we treat this as a secondary check. The controlled noise suite carries the main causal claim. No reported Recall@K result was found on Lacuna for PSG4D, so all PSG4D numbers will come from our own reruns.

The decisive checks are the following. Persistence should help most under temporally independent noise and on objects below 16 pixels, and its advantage over the $\alpha=0$ ablation should grow with noise severity. It should also help with fixed-in-image artifacts under camera motion. It should give no benefit under motion-coupled blur. If the gain does not shrink in that last family, the mechanism is not what we claim, and the result is more likely a regularization side effect.

Track-based warping should beat identity warping on fast-moving objects. Recall on newly appearing objects should drop for the first few frames by an amount predicted by $\alpha$ and recovered by the exploration budget. We will plot these new-object recall curves explicitly. Finally, wall-clock throughput and peak memory will be reported against TCDSG's roughly 20 frames per second as video resolution and length grow. This checks whether the fixed token budget delivers the claimed scaling on real hardware, given that TCFormer showed irregular token selection can erase theoretical savings.