Video Semantic Segmentation with Distortion-Aware Feature Correction

Jiafan Zhuang, Zilei Wang, Bingke Wang

Abstract

Video semantic segmentation has seen significant progress, yet per-frame processing remains computationally expensive for practical applications. To address this, many methods employ flow-based feature propagation to reuse features from previous frames. However, inaccuracies in optical flow estimation inevitably lead to distorted propagated features. In this paper, we propose a distortion-aware feature correction method to alleviate this issue. By transferring distortion patterns from the feature space to the image space, we enable effective distortion map prediction. Guided by these maps, our Feature Correction Module (FCM) rectifies features only in distorted areas. This approach significantly boosts accuracy at a low computational cost. Experimental results on Cityscapes and CamVid demonstrate that our method outperforms recent state-of-the-art techniques.

Introduction

Semantic segmentation—the task of assigning a category to every pixel—is a cornerstone of computer vision but is notoriously resource-intensive. In video contexts, processing every frame independently is often redundant. To improve efficiency, researchers typically use feature propagation: calculating optical flow between a key frame and the current frame to warp high-quality features forward.

The primary drawback of this approach is that optical flow is rarely perfect. Occlusions, rapid motion, and thin structures (like pedestrians or poles) cause "feature distortion." Previous attempts to fix these errors often relied on heavy networks that processed the entire frame, leading to unnecessary computation or "false correction" where perfectly good features were degraded. The challenge lies in performing "selective surgery"—fixing only the distorted regions without wasting resources on the rest of the frame.

Distortion-Aware Correction Framework

The proposed framework handles non-key frames through a three-part system designed for maximum efficiency:

  1. DMNet (Distortion Map Network): A lightweight Siamese network that compares the current frame with a warped version of the previous frame. It leverages "distortion consistency": if pixels are misaligned after warping, the underlying features are likely distorted as well. DMNet generates a distortion map ($M^D$) to pinpoint regions needing correction.
  2. CFNet (Correction Feature Network): A tiny network that extracts correction cues from the current frame. It is trained with a loss function weighted by the distortion map, ensuring it focuses its limited capacity on the most difficult, distorted areas.
  3. FCM (Feature Correction Module): This module performs a weighted fusion, retaining propagated features for stable regions and swapping in correction cues for distorted ones.
Visual comparison of propagated features versus corrected features using the proposed distortion-aware mechanism.

To stabilize training, the authors also introduced Dual Deep Supervision (DDS), which utilizes intermediate frames and pseudo-labels to provide additional learning guidance.

Experimental Results

The researchers evaluated their approach on three major benchmarks: Cityscapes, CamVid, and UAVid. The method consistently outperformed previous state-of-the-art models like DFF, DVSNet, and Accel.

Long-Distance Robustness

A significant breakthrough of this method is its stability during "long-distance propagation." Traditional models often see a sharp decline in accuracy as they move further from a key frame. Because the proposed method can effectively "patch" growing distortions, it maintains high accuracy even 9 frames away from the original key frame.

Segmentation results over a sequence of 9 frames, demonstrating stability during long-distance feature propagation.

Efficiency and Accuracy

On the Cityscapes dataset, the method achieved higher mean Intersection-over-Union (mIoU) scores while requiring significantly fewer GFLOPs than its competitors. By identifying distorted regions explicitly, the model avoids the computational overhead of a general overhaul, focusing its power only where it is needed most.

Conclusion and Future Work

The proposed distortion-aware correction provides a highly efficient path for video semantic segmentation. While the current DMNet is effective, the authors note a trade-off: using higher-level features for distortion prediction can further improve accuracy but at a higher computational cost. Future research may focus on balancing the complexity of the distortion detector with the correction network and exploring deeper temporal dependencies to further smooth results over extended video sequences.

Summary updated

Abstract

Video semantic segmentation is active in recent years benefited from the great progress of image semantic segmentation. For such a task, the per-frame image segmentation is generally unacceptable in practice due to high computation cost. To tackle this issue, many works use the flow-based feature propagation to reuse the features of previous frames. However, the optical flow estimation inevitably suffers inaccuracy and then causes the propagated features distorted. In this paper, we propose distortion-aware feature correction to alleviate the issue, which improves video segmentation performance by correcting distorted propagated features. To be specific, we firstly propose to transfer distortion patterns from feature into image space and conduct effective distortion map prediction. Benefited from the guidance of distortion maps, we proposed Feature Correction Module (FCM) to rectify propagated features in the distorted areas. Our proposed method can significantly boost the accuracy of video semantic segmentation at a low price. The extensive experimental results on Cityscapes and CamVid show that our method outperforms the recent state-of-the-art methods.
BibTeX
@article{zhuang2021video,
  author = {Jiafan Zhuang and Zilei Wang and Bingke Wang},
  title = {{Video Semantic Segmentation with Distortion-Aware Feature Correction}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2021},
  url = {https://openreview.net/forum?id=FrG7x0gPboW},
}
Figure