Temporally-Consistent Video Semantic Segmentation with Bidirectional Occlusion-guided Feature Propagation

Razieh Kaviani Baghbaderani, Yuanxin Li, Shuangquan Wang, Hairong Qi

Introduction

Semantic segmentation—the task of assigning a category label to every pixel—is a cornerstone of computer vision. While performance on static images has matured, video semantic segmentation (VSS) faces two significant hurdles: high computational costs and temporal inconsistency. Processing every frame independently often leads to "flickering" labels due to minor variations in lighting or noise.

Existing keyframe-based methods attempt to mitigate these issues by propagating features from a sparse set of high-quality "keyframes" to intermediate frames using optical flow. However, this approach struggles with occlusions and disocclusions, where objects moving behind one another create "ghosting" artifacts. The Bidirectional Occlusion-guided Feature Propagation (BOFP) framework addresses these challenges by leveraging information from both past and future keyframes.

Methodology

The BOFP framework moves beyond traditional unidirectional warping by implementing a three-pillar architecture designed for both efficiency and consistency.

The BOFP architecture overview, highlighting bidirectional propagation, OANet, and feature correction.

1. Bidirectional Feature Propagation

Instead of relying solely on the preceding keyframe ($I_k$), BOFP incorporates features from the subsequent keyframe ($I_{k+D}$). By warping features from both directions toward the current frame ($I_i$), the model can access visual information that might be occluded in one direction but visible in the other.

2. Occlusion-based Attention Network (OANet)

The core innovation of BOFP is the OANet. Rather than using rigid geometric rules to detect occlusions, OANet is a learnable module that processes four optical flow fields to generate soft attention maps ($A^f$ and $A^b$). These maps serve as a confidence score, indicating which regions of the warped features are reliable and which are likely distorted by motion or occlusion.

3. Occlusion-Guided Feature Correction

The final segmentation is a weighted fusion of three sources: forward-warped features, backward-warped features, and a "shallow" feature ($h_i^c$) extracted directly from the current frame. This shallow feature, generated by a lightweight network, is used specifically to fill in gaps where both warped versions are uncertain, typically around complex object boundaries.

Key Insights

  • The Power of Hindsight: Errors in forward propagation accumulate over time. By looking "back from the future," the model can correct these errors using information that was not yet available in the past.
  • Learning Uncertainty: By training OANet to predict the reliability of flow-based warping, the system becomes more robust to the inherent inaccuracies of optical flow algorithms.
  • Specialized Efficiency: BOFP uses a heavy model (SegNet_deep) only on sparse keyframes. For intermediate frames, it relies on efficient propagation and a lightweight "shallow" update, maintaining high accuracy at a fraction of the computational cost.

Results and Evaluation

The framework was evaluated on major benchmarks including Cityscapes, CamVid, and VSPW.

Comparison of forward, backward, and BOFP propagation. BOFP effectively eliminates artifacts (white boxes) present in unidirectional methods.

Temporal Consistency and Accuracy

BOFP significantly improves temporal stability. On the Cityscapes dataset, it achieved a mean Temporal Consistency (mTC) of 86.5%, surpassing previous state-of-the-art keyframe methods like DAVSS (84.5%). Qualitatively, the bidirectional approach eliminates the "ghosting" and flickering common in forward-only propagation, as shown in the visual comparisons above.

Performance Trade-offs

In terms of raw accuracy, BOFP achieved a mean Intersection over Union (mIoU) of 76.5% on Cityscapes while running at 9.6 FPS. While some per-frame models reach higher mIoU scores, they often operate at 1-2 FPS, making them unsuitable for real-time deployment. Ablation studies confirmed that adding the backward propagation branch alone improved accuracy by nearly 4%.

Conclusion and Future Work

BOFP provides a robust solution for temporally consistent video segmentation. While the requirement of a "future" keyframe introduces a small amount of latency—making it less ideal for zero-latency live streams—it is highly effective for buffered video processing and near-real-time applications like autonomous vehicle perception. Future research may focus on further optimizing the shallow feature extraction branch to bridge the remaining accuracy gap with per-frame models without sacrificing speed.

Summary updated

Abstract

Despite recent progress in static image segmentation, video segmentation is still challenging due to the need for an accurate, fast, and temporally consistent model. Conducting per-frame static image segmentation on a video is not acceptable since it is computationally prohibitive and prone to temporal inconsistency. In this paper, we present bidirectional occlusion-guided feature propagation (BOFP) method with the goal of improving temporal consistency of segmentation results without sacrificing segmentation accuracy, while at the same time keeping the operations at a low computation cost. It leverages temporal coherence in the video by feature propagation from keyframes to other frames along the motion paths in both forward and backward directions. We propose an occlusion-based attention network to estimate the distorted areas based on bidirectional optical flows, and utilize them as cues for correcting and fusing the propagated features. Extensive experiments on benchmark datasets demonstrate that the proposed BOFP method achieves superior performance in terms of temporal consistency while maintaining comparable level of segmentation accuracy at a low computation cost, striking a great balance among the three performance metrics essential to evaluate video segmentation solutions.
BibTeX
@inproceedings{baghbaderani2024temporally,
  author = {Razieh Kaviani Baghbaderani and Yuanxin Li and Shuangquan Wang and Hairong Qi},
  title = {{Temporally-Consistent Video Semantic Segmentation with Bidirectional Occlusion-guided Feature Propagation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00074},
}
Figure