Does Global Context Remove the Cost of Temporal Consistency Training?

When a segmentation network built for single images runs on video, its labels flicker from frame to frame even when nothing in the scene changes. A common fix is a temporal consistency loss, an extra training penalty that compares the prediction at frame $t$ with the prediction at frame $t-1$. The earlier prediction is first moved into the current frame's coordinates using optical flow, which is an estimate of where each pixel moved. This works, but it has a known cost. On real Cityscapes video, Varghese et al.'s unsupervised TC loss made HRNetV2 and DeepLabv3+ more stable, yet both lost some per-frame accuracy. The usual explanation is that flow is wrong around occlusions and motion boundaries, so the loss pushes correct predictions toward wrong targets.

We propose a second explanation that can be tested. Both networks in that study decide each pixel's label mostly from a local neighborhood of image texture. When such a network is told to stay stable under imperfect flow, the cheapest way to comply is to smooth local texture cues, and that costs accuracy. A network whose receptive field covers the whole image can instead stabilize on scene-level evidence, such as road layout or the extent of an object. Our claim is narrow. If the global-context module is trained end-to-end under the consistency loss, the accuracy cost of consistency training should shrink substantially. A local network with the same loss, the same flow, and the same parameter budget should keep paying that cost. The claim is also falsifiable: if masking out unreliable flow removes the cost for local networks just as well, the receptive-field explanation is wrong.

Core Claim

The accuracy–consistency trade-off is not a fixed property of flow-warped consistency losses. Its size depends on how much of the image each prediction can see. We expect three things. First, a global-context segmenter should reach a better Pareto frontier of per-frame accuracy (mIoU) against temporal consistency (mTC) than a local segmenter with matched parameters. Second, the advantage should come mainly from regions where flow is unreliable. Third, the advantage should vanish if the global module is frozen during consistency fine-tuning. The third prediction separates "global context helps" from "a stronger pretrained model helps."

This differs from the closest existing work in a specific way. Varghese et al. supply both the loss and the evidence of the trade-off, but they test only architectures with local or fixed-dilation context. MRCFA obtains consistency through cross-frame attention in the decoder and uses no consistency loss. Video Depth Anything pairs a global ViT backbone with a temporal loss, but it freezes that backbone, it needs ground-truth temporal gradients, and it predicts depth rather than class labels. None of these three trains a global spatial-routing module end-to-end with an unsupervised consistency loss on dense categorical prediction, and none asks whether doing so changes the trade-off.

Research Program

The global mechanism is CCNet's recurrent criss-cross attention. Each pixel attends along its row and column, and two passes connect every pixel to every other pixel. Its memory cost is low enough for full-resolution Cityscapes frames, about 11x less GPU memory than a dense non-local block.

A second global mechanism is needed to show the effect comes from receptive field and not from attention as such. For this we use ParCNetV2's oversized 1D convolutions, which reach a global receptive field inside a purely convolutional network. The local reference is DeepLabv3+, whose atrous pyramid gives sparse multi-scale context but no global routing. All three share one ResNet-scale encoder, and we match parameters and FLOPs within about 10%.

The loss follows Varghese et al. exactly. A supervised cross-entropy stage comes first. Fine-tuning then adds $\mathcal{L}_{CE} + \lambda \mathcal{L}_{TC}$, where $\mathcal{L}_{TC}$ is one minus the soft (differentiable) mIoU between the current probability map and the previous map warped by backward flow. Holding the loss fixed means any change in the trade-off can be attributed to the architecture. Every configuration is trained across a sweep of $\lambda$, so we compare whole accuracy–consistency curves rather than single operating points.

The analysis has two parts. First, we split each frame into flow-reliable and flow-unreliable pixels using a forward–backward flow consistency check. We then report where mIoU is lost and where flicker is removed. Second, we measure each model's effective receptive field before and after consistency fine-tuning. This tests whether the loss actually reshapes the global routing, the end-to-end adaptation that the frozen-backbone design of Video Depth Anything cannot provide.

A secondary arm swaps the soft-mIoU loss for a dense version of the MS-TCN truncated log-probability penalty. That penalty ignores frame-to-frame changes larger than a threshold, treating them as real boundaries rather than flicker. This arm asks whether truncation and global context are complementary or redundant ways of protecting true boundaries.

Baselines

The naive reference is each architecture trained with cross-entropy alone and evaluated frame by frame. It defines the starting accuracy and consistency that every other configuration must move away from.

The direct prior-method baseline is DeepLabv3+ with the Varghese TC loss, reproduced in our pipeline. Varghese et al. report a +2.78% absolute mTC gain for DeepLabv3+ and +4.25% for HRNetV2 on Cityscapes, with only "a slight trade-off" in mIoU and no exact number. Our reproduction must therefore measure the mIoU drop itself. We also add HRNetV2 with the TC loss, since it is the second local model they studied. Beating these baselines means that, for equal or higher mTC gain, the criss-cross model's mIoU drop relative to its own cross-entropy starting point is smaller, with non-overlapping confidence intervals over three seeds. Beating them only through a higher starting mIoU does not count.

The strongest existing method on dense video consistency is MRCFA. On VSPW with a SegFormer MiT-B1 backbone it reports 38.9% mIoU and gains of +4.1% mVC8 and +4.5% mVC16 over the next-best methods. These scores measure prediction agreement over windows of 8 and 16 frames. MRCFA pays for this in speed, dropping from about 58 FPS to 40 FPS. We will run our loss-trained criss-cross model on VSPW with the same MiT-B1 backbone and training schedule. A fair win is matching MRCFA's mVC within about one point at single-frame inference cost. Exceeding it at that cost would be a bonus.

On Cityscapes, CCNet's reported 81.9% test mIoU anchors per-frame accuracy, but it was obtained with a different backbone and schedule. We will treat it as context, not as a target.

Two ablations isolate the key ingredient. The first freezes the criss-cross module during consistency fine-tuning and trains only the rest of the network. The second replaces it with extra atrous branches of matched parameter count. If the frozen variant does as well as the trainable one, the claim about end-to-end adaptation fails. If the extra dilation does as well, the claim about global context fails.

Why not just…?

Why not just lower $\lambda$ until accuracy stops dropping? A smaller weight trades less consistency for less accuracy loss, which moves along the same curve rather than improving it. Our $\lambda$ sweep makes this explicit. The claim holds only if the global model's whole curve lies above the local model's. A better point on a single curve would not support it.

Why not just mask out pixels where flow is unreliable? This is the most serious alternative. It targets the flow-error explanation directly, and it may remove most of the accuracy cost on its own. We therefore apply forward–backward occlusion masking to both the local and global models as a full experimental factor. If masking closes the gap for DeepLabv3+, our receptive-field hypothesis is refuted and we will report it that way. If the global advantage persists under masking, or appears mainly in regions the mask cannot catch (such as consistent but wrong flow inside textureless surfaces), the hypothesis survives.

Why not just add more dilation or a bigger local backbone? DeepLabv3+ already uses an atrous pyramid and still showed the trade-off in Varghese et al.. MS-TCN++ found that large dilations alone lose fine detail and cause over-segmentation. The parameter-matched dilation ablation tests this directly in our setting.

Why not just get consistency from the architecture, as MRCFA does, or smooth predictions after the fact? MRCFA costs about a third of its throughput (58 to 40 FPS) and still leaves residual flicker, since no loss targets it. Post-hoc filters blur real boundaries, which is the failure that motivated truncated losses in MS-TCN. Both are in our comparison set at equal backbone.

Proposed Experiments

The primary benchmark is Cityscapes video sequences. Accuracy is mIoU on the sparsely labeled frames, and stability is the mTC metric from Varghese et al. on unlabeled neighboring frames. Sim KI-A provides labels on every frame, so it tests whether consistency gains coincide with real accuracy gains rather than stable errors. Varghese et al. report +2.19% mIoU and +0.21% mTC there for DeepLabv3+. VSPW provides the head-to-head comparison with MRCFA on mVC8 and mVC16.

The decisive check is a single statistic. For each architecture we compute the mIoU change caused by consistency training at matched mTC gain, then test the interaction between architecture and occlusion masking. The proposal succeeds if three conditions all hold:

  1. The global models lose significantly less mIoU than DeepLabv3+.
  2. The difference survives occlusion masking.
  3. The difference disappears when the global module is frozen.

It fails if masking alone equalizes the local and global models, and we will report that outcome.