Resolution-Routed Supervision for Synthetic-to-Real Video Segmentation

Networks that keep full-resolution feature maps through every layer are the most reliable way to segment thin, small, and distant objects such as poles, signs, bicycle frames, and far-away pedestrians. The cheapest way to get pixel-perfect video labels for such objects is a game engine. These two facts conflict. The fine detail a high-resolution network is built to keep is the same band of the signal where rendered images differ most from real camera images. So the better the network preserves detail, the more synthetic texture it memorizes. Real driving video has the right fine-scale statistics, but it is labeled only sparsely, on one frame in five in the standard benchmark.

We propose to route each supervision source to the resolution where it is trustworthy. Dense synthetic video trains the coarse, semantic parts of a multi-resolution network. The full-resolution stream never receives gradients from synthetic data. Instead it is trained on real video through a keyframe-to-keyframe transport loss: a label on real frame $t$ is carried through the unlabeled real frames by the network's own full-resolution feature correspondences, and must arrive matching the label on frame $t+k$. The fine stream thus learns real-world boundaries from the real frames between annotations. Because both ends of each chain are labeled, drift is penalized directly rather than accumulated.

Core Claim

In synthetic-to-real video segmentation with sparse real labels, the synthetic-to-real gap in a high-resolution network sits mostly in its finest stream. That stream can be trained adequately from sparsely labeled real video alone, provided the unlabeled intermediate frames are used as a correspondence signal anchored at labeled endpoints. The prediction is specific and testable. Relative to the usual recipe of pretraining on synthetic data and fine-tuning on real keyframes, resolution routing should raise video panoptic quality mainly on thin and small classes and in narrow bands around object boundaries. It should also make the fine-stream features harder for a synthetic-versus-real classifier to tell apart. Blocking synthetic gradients without the transport loss should not produce these gains, and neither should the transport loss without the blocking.

Where This Sits

The tension comes from two lines of evidence.

On the high-resolution side, Lite-HRNet keeps parallel high-to-low resolution streams that exchange information throughout the network. At 0.7 GFLOPs it reaches 70.4 AP on COCO pose, against 67.8 AP for MobileNetV2 at 3.33 GFLOPs, which shows full-resolution streams are affordable. Deep Interactive Thin Object Selection shows one-pixel-wide structures vanish under standard striding. Both motivate keeping a full-resolution stream.

On the sim-to-real side, MRFP finds that the highest- and lowest-frequency features carry the most domain variance, and that mid-level structure transfers. It counters this by randomly perturbing fine and global feature statistics during synthetic training. On static images it raises mean mIoU across five real targets from 31.21% to 39.27% with ResNet-50. MRFP both motivates and constrains our design: it locates the gap where our high-resolution stream lives. It works only on single frames and never uses real data during training. It suppresses synthetic fine texture but offers nothing to replace it with. We replace it with real fine structure taken from real video.

Closest in setting are VPSNet and Slot-VPS. Both use the synthetic VIPER video dataset and are evaluated on Cityscapes-VPS, which is labeled every fifth frame. VPSNet aligns features across frames with optical flow. Slot-VPS replaces this with object-centric "slots" that compete across frames. Neither adapts across domains, and both use stride-32 backbones. We keep their benchmark but change what each data source is allowed to teach.

DOVTrack fills 30-frame label gaps by running a feature-level diffusion model between object embeddings. It operates on pooled box vectors, so it discards the spatial layout our proposal depends on. AUSM supplies supporting evidence that keeping past masks as spatial maps rather than compressed vectors helps video segmentation, but it assumes dense-ish real labels and no synthetic source.

On the sparse-label side, Bai et al. propagate keyframe labels with non-rigid registration and have to down-weight the propagated labels exponentially with distance ($e^{-0.1\Delta t}$) because errors accumulate. That negative result drives our choice to anchor chains at both labeled ends rather than propagate outward from one. CSTM matches features patch by patch across sparse cardiac keyframes. It uses real data only, and the matching is used at inference rather than as a training signal for the fine features.

Research Program

The model is a parallel multi-resolution backbone in the style of Lite-HRNet, with streams at strides 4, 8, 16, and 32 and cross-stream fusion. It feeds a panoptic head with a temporal association module adapted from Slot-VPS, so the benchmark protocol matches the strongest prior system. Training alternates synthetic and real batches.

Synthetic batches. On synthetic VIPER clips, the full panoptic and tracking losses apply. A stop-gradient is placed wherever the stride-4 stream enters or leaves the fusion units. Synthetic gradients therefore update the coarser streams and the head but never the stride-4 weights. At inference the head still reads the stride-4 stream, so the head must learn to use features it did not shape on synthetic data.

Real batches. On real Cityscapes-VPS clips, the ordinary panoptic loss applies on labeled keyframes to all streams. The transport loss adds the signal from the unlabeled frames. Let $f_i$ be the stride-4 feature map of frame $i$. Let $A_{i\to i+1}$ be a local affinity matrix: a softmax over feature similarity within a window of radius $r$ around each pixel, so memory stays linear in image size. Let $y_t$ be the one-hot panoptic label at keyframe $t$. The transported label and loss are

$$ \hat y_{t+k} = A_{t+k-1\to t+k}\cdots A_{t\to t+1}\,y_t,\qquad \mathcal{L}_{\text{transport}} = \mathrm{CE}(\hat y_{t+k},\,y_{t+k}) + \mathrm{CE}(\hat y_{t}^{\leftarrow},\,y_{t}), $$

where $\hat y_t^{\leftarrow}$ is the same chain run backward from $y_{t+k}$, and $k=5$ on Cityscapes-VPS. The loss flows only into the stride-4 stream. It rewards fine features that correspond correctly across real motion, blur, and occlusion over the full gap, which is exactly what boundary tracking needs. Because the endpoint is ground truth, the drift that forced Bai et al. to discount distant frames becomes the quantity being minimized.

Analysis. We will check three things. First, a frequency analysis of each stream's features, tracking domain-classifier accuracy per stream before and after routing. Second, per-class and boundary-band error breakdowns. Third, a label-sparsity sweep. For the sweep we subsample Cityscapes-VPS keyframes to every 10th and every 15th frame (chaining over the dropped labels) and to a single labeled frame per snippet, which leaves no second endpoint and so falls back to a cycle loss.

Baselines

Naive reference. The same high-resolution network, trained only on VIPER and evaluated directly on Cityscapes-VPS. This measures the raw gap and how much of it the fine stream causes. We compute that share by swapping in a fine stream trained on real data.

Standard practice. The same network pretrained on VIPER and then fine-tuned on real keyframes with the ordinary panoptic loss, using identical schedules, data, and GPU-hours. Most practitioners would do this. The comparison isolates everything our method adds.

Strongest reported system. Slot-VPS reports 63.7 VPQ on Cityscapes-VPS with ResNet-50 at 4.2 FPS, above VPSNet's 56.1 VPQ at 1.0 FPS. This is the number to beat. Its backbone and schedule differ from ours, so we will also retrain Slot-VPS under our data and compute budget. We will report both its published figure and our rerun, and claim improvement only against the rerun at matched compute.

Strongest sim-to-real regularizer. MRFP+ perturbation inserted into our high-resolution network during synthetic training, followed by the same real fine-tuning. MRFP's reported 39.27% mean mIoU is a static, image-level result on a different protocol, so it is not directly comparable to VPQ. No reported result was found on Lacuna for MRFP applied to video panoptic segmentation. Its perturbation strength is tuned with the same search budget as our loss weight and window radius.

Flow-based label filling. Intermediate real frames receive labels warped by optical flow from the nearest keyframe and are used as ordinary targets. This follows the spirit of VPSNet's fuse module and Bai et al.'s registration labels, with their distance decay.

Ablations. Two ablations isolate the key ingredients. The first adds the transport loss to all streams while synthetic gradients still reach the fine stream: this is temporal self-supervision without routing. The second applies routing without the transport loss: the fine stream is starved of synthetic gradients but learns only from real keyframes.

What counts as beating a baseline. A win means higher VPQ at every window size ($k=5,10,15$) averaged over three seeds, with most of the gain on thin and small classes and in the boundary band. VPQ gains confined to large "stuff" classes such as road and sky would count against the claim.

Why not just…?

Why not just pretrain on VIPER and fine-tune everything on real keyframes? This is the real threat. Fine-tuning may already overwrite the synthetic fine texture, since the stride-4 stream sees real keyframes too. Our counter-argument is that about 3,000 real labeled frames are few relative to what the fine stream has absorbed from dense synthetic video. Also, only one frame in five contributes to fine-tuning, so the real motion statistics in the other frames go unused. The standard-practice baseline together with the per-stream domain-classifier probe settles this. If fine-tuning alone drives fine-stream domain accuracy to chance and matches our VPQ on thin classes, routing is unnecessary and we will say so.

Why not just apply MRFP-style perturbation to the high-resolution stream? MRFP stops the network relying on synthetic fine texture, but it gives the fine stream no real signal to learn from. On static images its gains come without any target data. We expect perturbation to help the naive model but to lag behind routing once real video is available. The MRFP baseline, run under the same budget, tests this. We will also try perturbation combined with routing: if the two gains add, the mechanisms are complementary rather than redundant.

Why not just fill the gaps with optical-flow pseudo-labels? Flow warping is simple. It uses the intermediate real frames, and five-frame gaps in driving video are modest. But Bai et al. needed exponential down-weighting because warped labels degrade with distance. VPSNet's reliance on flow also costs speed, and Slot-VPS beat it by dropping flow. Our transport loss differs in two ways: correspondences come from the network's own fine features, which it trains, and every chain ends at ground truth. The flow baseline, and especially the sparsity sweep to 10- and 15-frame gaps, will show whether this matters. We expect flow to be competitive at $k=5$ and to fall behind as the gap widens.

Why not just add temporal self-supervision without blocking synthetic gradients? If the transport loss alone captures the benefit, routing is ornamental. The no-routing ablation tests exactly this. The claim stands only if routing plus transport beats transport alone on thin classes.

Proposed Experiments and Decisive Checks

The main experiment trains on VIPER plus Cityscapes-VPS at its native one-in-five labeling. We report VPQ at $k=5,10,15$, image-level panoptic quality, boundary-band IoU in 3- and 8-pixel bands, per-class VPQ for pole, traffic sign, rider, bicycle, and motorcycle, identity switches, and throughput against Slot-VPS's 4.2 FPS. The sparsity sweep repeats every method at each label density.

Three checks decide the proposal. The first is localization: gains must concentrate in thin classes and boundary bands, not in large regions. The second is mechanism: under routing, a linear synthetic-versus-real probe on stride-4 features must fall toward chance, while probes on the coarse streams stay above chance, consistent with MRFP's finding that coarse style is also domain-variant. The third is robustness to sparsity: the margin over flow-based filling must hold or widen as labeled frames thin out.

If routing matches plain fine-tuning on all three checks, we conclude that real keyframe fine-tuning already repairs the fine stream. That would be a useful negative result, telling practitioners to spend synthetic data freely at every resolution.