Introduction
Semantic segmentation—the task of assigning a category label to every pixel—is a cornerstone of computer vision. While performance on static images has matured, video semantic segmentation (VSS) faces two significant hurdles: high computational costs and temporal inconsistency. Processing every frame independently often leads to "flickering" labels due to minor variations in lighting or noise.
Existing keyframe-based methods attempt to mitigate these issues by propagating features from a sparse set of high-quality "keyframes" to intermediate frames using optical flow. However, this approach struggles with occlusions and disocclusions, where objects moving behind one another create "ghosting" artifacts. The Bidirectional Occlusion-guided Feature Propagation (BOFP) framework addresses these challenges by leveraging information from both past and future keyframes.
Methodology
The BOFP framework moves beyond traditional unidirectional warping by implementing a three-pillar architecture designed for both efficiency and consistency.
1. Bidirectional Feature Propagation
Instead of relying solely on the preceding keyframe ($I_k$), BOFP incorporates features from the subsequent keyframe ($I_{k+D}$). By warping features from both directions toward the current frame ($I_i$), the model can access visual information that might be occluded in one direction but visible in the other.
2. Occlusion-based Attention Network (OANet)
The core innovation of BOFP is the OANet. Rather than using rigid geometric rules to detect occlusions, OANet is a learnable module that processes four optical flow fields to generate soft attention maps ($A^f$ and $A^b$). These maps serve as a confidence score, indicating which regions of the warped features are reliable and which are likely distorted by motion or occlusion.
3. Occlusion-Guided Feature Correction
The final segmentation is a weighted fusion of three sources: forward-warped features, backward-warped features, and a "shallow" feature ($h_i^c$) extracted directly from the current frame. This shallow feature, generated by a lightweight network, is used specifically to fill in gaps where both warped versions are uncertain, typically around complex object boundaries.
Key Insights
- The Power of Hindsight: Errors in forward propagation accumulate over time. By looking "back from the future," the model can correct these errors using information that was not yet available in the past.
- Learning Uncertainty: By training OANet to predict the reliability of flow-based warping, the system becomes more robust to the inherent inaccuracies of optical flow algorithms.
- Specialized Efficiency: BOFP uses a heavy model (
SegNet_deep) only on sparse keyframes. For intermediate frames, it relies on efficient propagation and a lightweight "shallow" update, maintaining high accuracy at a fraction of the computational cost.
Results and Evaluation
The framework was evaluated on major benchmarks including Cityscapes, CamVid, and VSPW.
Temporal Consistency and Accuracy
BOFP significantly improves temporal stability. On the Cityscapes dataset, it achieved a mean Temporal Consistency (mTC) of 86.5%, surpassing previous state-of-the-art keyframe methods like DAVSS (84.5%). Qualitatively, the bidirectional approach eliminates the "ghosting" and flickering common in forward-only propagation, as shown in the visual comparisons above.
Performance Trade-offs
In terms of raw accuracy, BOFP achieved a mean Intersection over Union (mIoU) of 76.5% on Cityscapes while running at 9.6 FPS. While some per-frame models reach higher mIoU scores, they often operate at 1-2 FPS, making them unsuitable for real-time deployment. Ablation studies confirmed that adding the backward propagation branch alone improved accuracy by nearly 4%.
Conclusion and Future Work
BOFP provides a robust solution for temporally consistent video segmentation. While the requirement of a "future" keyframe introduces a small amount of latency—making it less ideal for zero-latency live streams—it is highly effective for buffered video processing and near-real-time applications like autonomous vehicle perception. Future research may focus on further optimizing the shallow feature extraction branch to bridge the remaining accuracy gap with per-frame models without sacrificing speed.
Summary updated