Video dense prediction asks a network to label every pixel of every frame, often for several tasks at once: what each pixel is (semantic segmentation), how far away it is (depth), where it moves (optical flow), and which object it belongs to over time (tracking). Some of these tasks are about properties that do not change within a clip, such as the road being road and the building being at roughly the same 3D place. Others are about what changes between frames. Today's multi-task dense predictors, the networks with one shared trunk and several task-specific output heads, process each frame with a single entangled feature map. Video systems then bolt on an optical-flow network to carry information across time.
We propose to split the representation at every resolution of the feature pyramid into two parts. The first is a small bank of clip-level "static" tokens, built so that they cannot depend on frame order. The second is a per-frame spatial "dynamic" map that says where things are right now. Each task head reads from both through attention, and a learned gate per task and per scale records how much it relies on each. The bet is threefold: time-invariant tasks become temporally consistent without flow warping, motion tasks stop fighting them for the same features, and the gates give the first measurement of how static-versus-dynamic dependence changes with scale.
Core Claim
Sequential disentanglement has long split a sequence into one global static code and a sequence of per-step dynamic codes. Disentangled Sequential Autoencoder introduced this split with a flat global latent and a recurrent dynamic latent. Sequential Representation Learning via Static-Dynamic Conditional Disentanglement enforced time-invariance by shuffling frames and requiring the same reconstruction, and let dynamics depend on the static code. DiffSDA did the same with a diffusion decoder. All three work on clean, single-object data with flat vectors and a single generative objective.
Multi-task dense prediction has gone the other way. MTI-Net shows that task interactions are scale-dependent across four resolutions. BridgeNet and DeMT refine how task features interact. All three work only on single images.
Our claim is that the static/dynamic split becomes useful for dense video prediction only if it is made multi-scale and egomotion-tolerant. Making it multi-scale is the role of hierarchical fusion. Making it egomotion-tolerant is why the static part must be a set of tokens rather than a pixel-aligned map. Once both hold, it acts as a structural fix for multi-task conflict between time-invariant and time-varying tasks. The pixel-alignment point is what distinguishes this from the obvious combination. In driving video the camera moves, so a "static" feature map averaged over time in image coordinates is simply blurred. The static content has to be stored as permutation-invariant clip-level tokens that each frame's dynamic map queries to recover what is where.
Research Program
The model uses a shared image backbone producing features at 1/4, 1/8, 1/16 and 1/32 resolution, the scales MTI-Net uses.
At each scale $s$, a small set of learned queries attends jointly over the features of all $T$ frames in the clip and produces $K_s$ static tokens. Pooling across frames is order-free by construction. We add the shuffle constraint borrowed from Static-Dynamic Conditional Disentanglement: static tokens computed from a random subset of frames, in random order, must produce the same task predictions as those from the full clip. In parallel, each frame keeps a narrow dynamic map at that scale. The narrowness is the information-bottleneck argument from the Disentangled Sequential Autoencoder: the dynamic channel is too small to re-encode scene content, so content is pushed into the static tokens.
Following the evidence in the Static-Dynamic Conditional Disentanglement paper and DiffSDA that strict independence causes leakage, we do not make the two parts independent. The dynamic map at scale $s$ is computed with cross-attention to that scale's static tokens, so motion features can depend on scene structure.
Each task head fuses scales coarse-to-fine, as MTI-Net's propagation module does. At every scale it forms its input as a gated sum of the dynamic map and the static-token readout, where the readout is the dynamic map attending to the static tokens. The gate $g_{t,s}\in[0,1]$ is learned per task $t$ and scale $s$. These gates are the analysis instrument. Our hypothesis, from the scale dependence MTI-Net found for cross-task affinity, is that segmentation and depth draw heavily on static tokens at coarse scales but less at fine scales. Flow should rely on the dynamic map at every scale. Tracking should mix both, with static tokens supplying object identity and the dynamic map supplying position. That would be the first direct measurement of how static-dynamic coupling varies with scale.
Training is standard joint multi-task training with fixed loss weights. We avoid a pretrain-then-freeze recipe because On the relationship between disentanglement and multi-task learning found mixed gains when disentangled latents were pretrained and then fed to task heads. The same work found that joint hard-parameter sharing itself produced more disentangled codes. That paper motivates joint training, and it constrains us by warning that disentanglement by itself is not a guaranteed multi-task win.
Baselines
The naive reference point is a per-frame single-task network for each task on the same backbone and training schedule. It defines the multi-task improvement $\Delta_m$, the average relative change over single-task models that MTI-Net and DeMT report.
The strongest static-image competitor is MTI-Net run per frame, extended with a temporal module. For Cityscapes-VPS only, we also run BridgeNet per frame. MTI-Net reported a +10.91% $\Delta_m$ over single-task baselines on NYUD-v2, and its multi-scale distillation is exactly what our model adds to. It gets the same backbone, clip length, data and hyperparameter search budget as ours.
The strongest video competitor is VPSNet from Video Panoptic Segmentation, which fuses features across frames using optical flow. It reports 56.1 Video Panoptic Quality (VPQ) on Cityscapes-VPS and 62.2 Panoptic Quality on static Cityscapes. Beating it means exceeding 56.1 VPQ on Cityscapes-VPS with no external flow network, or matching it at clearly lower runtime.
On Virtual KITTI 2 we found no reported multi-task dense result on Lacuna. The dataset paper reports per-task references only, such as 91.2% Multi-Object Tracking Accuracy (MOTA) for a dedicated tracker, so all Virtual KITTI 2 baselines will be trained by us under one protocol.
The decisive ablation is a model with unrestricted cross-frame attention at every scale. It has the same parameter count and compute as ours but no static token bottleneck, no shuffle constraint and no gates. It isolates whether the static/dynamic structure matters or whether temporal attention capacity alone explains the gains. A second ablation enforces strict independence by removing the dynamic-to-static cross-attention, which tests the leakage argument in a dense-prediction setting. A third replaces the token bank with a pixel-aligned temporally averaged map, which tests the egomotion argument.
Beating the baselines means a higher $\Delta_m$ than both per-frame MTI-Net and the cross-frame-attention ablation, with no task worse than its single-task model. It also means lower temporal flicker on static pixels, measured as the disagreement between predictions warped with ground-truth flow.
Why not just…?
Why not just run MTI-Net or BridgeNet frame by frame? They are strong on images but carry no information across time. Every frame re-derives scene content, so predictions flicker and tracking gets no identity signal. The per-frame baseline measures this directly. If it matches our $\Delta_m$ and flicker, the idea fails on Virtual KITTI 2.
Why not just add cross-frame attention at every scale? This is the real threat, because our token bank is a constrained version of it. Unconstrained attention lets flow and segmentation share the same entangled temporal features, which is where we expect negative transfer to remain. Unconstrained attention also gives no reason for representations to be order-invariant. The matched-compute ablation settles accuracy. Two further measurements settle the rest: the cosine similarity between task gradients on shared parameters, and robustness when frames are dropped or reordered at test time. If the ablation ties on all three, we will report that the structure adds interpretability and nothing more.
Why not just warp features with optical flow, as VPSNet does? Flow warping costs a separate network at training and test time. It also carries only instantaneous pixel motion, so identity can drift through occlusions. Our model has no flow input. The comparison on Cityscapes-VPS (VPQ and runtime) and the occlusion slices of Virtual KITTI 2 tracking will show whether clip-level tokens can replace warping.
Why not just pretrain a sequential disentanglement model and attach heads? The evidence on Lacuna argues against this. On the relationship between disentanglement and multi-task learning found mixed downstream gains from pretrained disentangled latents. Existing sequential disentanglers such as DiffSDA also use flat codes and slow iterative decoders unsuited to dense real-time heads. We include a frozen-pretrained-token variant to confirm this.
Proposed Experiments
The primary testbed is Virtual KITTI 2. It provides semantic segmentation, depth, optical flow and tracking labels on the same clips, so all four tasks can be trained jointly. We report mean intersection-over-union, depth root-mean-square error, flow endpoint error, MOTA, $\Delta_m$ and static-pixel flicker.
Virtual KITTI 2 also enables a disentanglement test that ordinary dense benchmarks lack, which addresses the missing factor-level evaluation in complex scenes noted across the disentanglement literature. The dataset renders the same scenes under clear, fog and rain conditions and under camera rotations. We compute static tokens from the clear-weather clone and feed them to the dynamic maps of the foggy clone. If the static tokens encode scene content rather than appearance noise, foggy-frame segmentation and depth should improve. The reverse swap, with fog-derived tokens used on clear frames, should degrade predictions only mildly. For rotated-camera clones, the dynamic maps should absorb the rotation while the static tokens stay nearly unchanged, which we measure by token similarity across clones.
The secondary benchmark is Cityscapes-VPS, where we compare against VPSNet's reported 56.1 VPQ.
A short sanity run on LPCSprites checks that the token-bank mechanism still recovers body-type identity under content swapping, without claiming the 99.97% accuracy reported by the Static-Dynamic Conditional Disentanglement paper. This confirms that the mechanism disentangles before we trust it at scale.
The study has a decisive outcome on each of its claims. The architecture claim holds or fails on $\Delta_m$ and flicker against the matched cross-frame-attention ablation. The disentanglement claim holds or fails on the cross-weather token swap. The scale-dependence claim holds or fails on whether the learned gates show the predicted coarse-static, fine-dynamic pattern for segmentation and depth, consistently across random seeds.
References
- Disentangled Sequential Autoencoder
- Sequential Representation Learning via Static-Dynamic Conditional Disentanglement
- DiffSDA: Unsupervised Diffusion Sequential Disentanglement Across Modalities
- MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning
- BridgeNet: Comprehensive and Effective Feature Interactions via Bridge Feature for Multi-Task Dense Predictions
- DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction
- On the Relationship Between Disentanglement and Multi-task Learning
- Video Panoptic Segmentation
- Virtual KITTI 2