Abstract
Video semantic segmentation has seen significant progress, yet per-frame processing remains computationally expensive for practical applications. To address this, many methods employ flow-based feature propagation to reuse features from previous frames. However, inaccuracies in optical flow estimation inevitably lead to distorted propagated features. In this paper, we propose a distortion-aware feature correction method to alleviate this issue. By transferring distortion patterns from the feature space to the image space, we enable effective distortion map prediction. Guided by these maps, our Feature Correction Module (FCM) rectifies features only in distorted areas. This approach significantly boosts accuracy at a low computational cost. Experimental results on Cityscapes and CamVid demonstrate that our method outperforms recent state-of-the-art techniques.
Introduction
Semantic segmentation—the task of assigning a category to every pixel—is a cornerstone of computer vision but is notoriously resource-intensive. In video contexts, processing every frame independently is often redundant. To improve efficiency, researchers typically use feature propagation: calculating optical flow between a key frame and the current frame to warp high-quality features forward.
The primary drawback of this approach is that optical flow is rarely perfect. Occlusions, rapid motion, and thin structures (like pedestrians or poles) cause "feature distortion." Previous attempts to fix these errors often relied on heavy networks that processed the entire frame, leading to unnecessary computation or "false correction" where perfectly good features were degraded. The challenge lies in performing "selective surgery"—fixing only the distorted regions without wasting resources on the rest of the frame.
Distortion-Aware Correction Framework
The proposed framework handles non-key frames through a three-part system designed for maximum efficiency:
- DMNet (Distortion Map Network): A lightweight Siamese network that compares the current frame with a warped version of the previous frame. It leverages "distortion consistency": if pixels are misaligned after warping, the underlying features are likely distorted as well. DMNet generates a distortion map ($M^D$) to pinpoint regions needing correction.
- CFNet (Correction Feature Network): A tiny network that extracts correction cues from the current frame. It is trained with a loss function weighted by the distortion map, ensuring it focuses its limited capacity on the most difficult, distorted areas.
- FCM (Feature Correction Module): This module performs a weighted fusion, retaining propagated features for stable regions and swapping in correction cues for distorted ones.
To stabilize training, the authors also introduced Dual Deep Supervision (DDS), which utilizes intermediate frames and pseudo-labels to provide additional learning guidance.
Experimental Results
The researchers evaluated their approach on three major benchmarks: Cityscapes, CamVid, and UAVid. The method consistently outperformed previous state-of-the-art models like DFF, DVSNet, and Accel.
Long-Distance Robustness
A significant breakthrough of this method is its stability during "long-distance propagation." Traditional models often see a sharp decline in accuracy as they move further from a key frame. Because the proposed method can effectively "patch" growing distortions, it maintains high accuracy even 9 frames away from the original key frame.
Efficiency and Accuracy
On the Cityscapes dataset, the method achieved higher mean Intersection-over-Union (mIoU) scores while requiring significantly fewer GFLOPs than its competitors. By identifying distorted regions explicitly, the model avoids the computational overhead of a general overhaul, focusing its power only where it is needed most.
Conclusion and Future Work
The proposed distortion-aware correction provides a highly efficient path for video semantic segmentation. While the current DMNet is effective, the authors note a trade-off: using higher-level features for distortion prediction can further improve accuracy but at a higher computational cost. Future research may focus on balancing the complexity of the distortion detector with the correction network and exploring deeper temporal dependencies to further smooth results over extended video sequences.
Summary updated