Video Semantic Segmentation With Distortion-Aware Feature Correction

Jiafan Zhuang, Zilei Wang, Bingke Wang

TL;DR

Video semantic segmentation often relies on "feature propagation" to save power, but inaccurate optical flow frequently distorts these features. This paper introduces a Distortion-Aware Feature Correction (DAVSS) method that uses a lightweight network to identify distorted regions and surgically correct them, achieving state-of-the-art accuracy and speed by focusing only on what’s broken.

What Problem Does This Paper Address?

Semantic segmentation—assigning a category to every pixel in an image—is computationally expensive. In video, processing every frame from scratch is often overkill because consecutive frames are highly redundant. To speed things up, researchers use feature propagation: they calculate the "optical flow" (motion) between a key frame and the current frame, then warp the high-quality features from the key frame to fit the new one.

The catch? Optical flow is rarely perfect. Occlusions, fast motion, and thin objects (like poles or pedestrians) lead to "feature distortion." Previous attempts to fix this either used heavy, expensive networks to refine the entire frame or applied simple scaling that often ruined perfectly good features—a phenomenon the authors call false correction. The challenge is to fix the distorted parts of a frame without wasting computation on the parts that are already correct.

Methods

The authors propose a three-part framework to handle non-key frames efficiently:

  1. DMNet (Distortion Map Network): This is a ultra-lightweight Siamese network. It compares the current video frame with a "warped" version of the previous frame. The key insight here is distortion consistency: if the pixels in the image look misaligned after warping, the underlying features are likely distorted too. DMNet produces a "distortion map" ($M^D$) that identifies exactly which regions need help.
  2. CFNet (Correction Feature Network): Instead of a full-blown segmentation model, the authors use a tiny network to extract "correction cues" from the current frame. Crucially, this network is trained using a loss function weighted by the distortion map, forcing it to ignore easy areas and focus on the difficult, distorted regions.
  3. FCM (Feature Correction Module): This module performs a weighted fusion. It keeps the propagated features for "safe" regions and swaps in the correction cues for distorted regions.

To make training more stable, they also introduced Dual Deep Supervision (DDS), which adds an intermediate frame and uses "pseudo-labels" to provide extra guidance during the learning process.

Datasets

The researchers evaluated their approach on three major benchmarks:

  • Cityscapes: The gold standard for urban street scene understanding.
  • CamVid: A smaller but widely used dataset of automotive video.
  • UAVid: A challenging high-resolution dataset captured from drones, featuring significant scale variation and moving objects.

Key Insights and Ideas

  • The "Aha!" of Distortion Consistency: You don't need a heavy deep network to know where features are broken. By simply looking at how well the images align after a warp, the model can accurately predict where the features are failing.
  • Selective Surgery vs. General Overhaul: Most previous models (like Accel or DFF) treat every pixel the same. By explicitly identifying distorted regions, DAVSS avoids "false correction," ensuring that high-quality features from the key frame aren't accidentally degraded by a lightweight "fixer" network.
  • Efficiency through Sparsity: Because the correction network only needs to be good at fixing specific errors (rather than understanding the whole scene from scratch), it can be designed to be extremely small, keeping the overall "price" of the correction low.

Key Results

The proposed method consistently outperformed previous state-of-the-art models like DFF, DVSNet, and Accel across all datasets.

  • Accuracy vs. Speed: On the Cityscapes dataset, DAVSS achieved higher mean Intersection-over-Union (mIoU) scores while using significantly fewer GFLOPs (computational units) than competitors.
  • Long-Distance Robustness: One of the biggest wins was in "long-distance propagation." While other models' accuracy plummeted as they moved further away from a key frame (e.g., 9 frames later), DAVSS remained much more stable because it could effectively "patch" the growing distortions.
  • Upper Bound Analysis: The authors showed that their method's performance is very close to the "theoretical limit" (using ground-truth distortion maps), suggesting their DMNet is highly effective at identifying errors.

Limitations and Future Work

While the method is highly efficient, the authors noted a trade-off in the Design of Distortion Map Prediction. Using higher-level features to predict the distortion map yields even better accuracy but increases the computational cost. Finding the "perfect" balance between the complexity of the distortion detector and the correction network remains an open area.

Additionally, while the method handles spatial distortion well, future work could explore more complex temporal dependencies to further smooth out segmentation results over very long video sequences.

Summary updated

Abstract

Video semantic segmentation aims to generate an accurate semantic map for each frame in a video. For such a task, conducting per-frame image segmentation is generally unacceptable in practice due to high computation cost. To address this issue, many works perform the flow-based feature propagation to reuse the features of previous frames, which essentially exploits the content continuity of consecutive frames. However, the estimated optical flow would inevitably suffer inaccuracy and then make the propagated features distorted. In this article, we propose a distortion-aware feature correction method with the goal of improving video segmentation performance at a low price. Our core idea is to correct the features on distorted regions using the current frame while reserving the propagated features for other regions. In this way, a lightweight network is enough for achieving promising segmentation results. In particular, we propose to predict the distorted regions by utilizing the consistency of distortion patterns in images and features, such that the high-cost feature extraction from current frames can be avoided. We conduct extensive experiments on Cityscapes, CamVid, and UAVid, and the results show that our proposed method significantly outperforms previous methods and achieves the state-of-the-art performance on both segmentation accuracy and speed. Code and pretrained models are available at https://github.com/jfzhuang/DAVSS.
BibTeX
@article{zhuang2021video,
  author = {Jiafan Zhuang and Zilei Wang and Bingke Wang},
  title = {{Video Semantic Segmentation With Distortion-Aware Feature Correction}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2021},
  url = {https://openreview.net/forum?id=FrG7x0gPboW},
}

Loading related content...

Figure