Attention-Guided Network for Semantic Video Segmentation

Jiangyun Li, Yikai Zhao, Jun Fu, Jiajia Wu, Jing LiuIEEE Access 2019 · OpenReview

Abstract

Remarkable success has been made by deep convolutional neural network (CNN) models in semantic image segmentation. However, most segmentation models are based on classification networks which tend to learn image-level features and lost abundant spatial information due to repeated pooling and downsampling operations, and the CNN-based methods are not robust to inputs, hence directly applying existing segmentation methods to semantic video segmentation will result in spatially inconsecutive and temporally inconsistent segmentation predictions within one instance and of the same objects across adjacent frames, respectively. To tackle this challenge, we propose an Attention-Guided Network (AGNet) to adaptively strengthen inter-frame and intra-frame features for more precise segmentation predictions. Specifically, we append an adjacent attention module (AAM) and a spatial attention module (SAM) on the top of dilated fully convolutional network (FCN), which model the feature correlations in temporal and spatial dimensions, respectively. The AAM selectively enhances the inter-frame features of the same objects across adjacent frames for temporally consistent predictions. Meanwhile, the SAM selectively aggregates the intra-frame features within one instance for spatially consecutive predictions. Finally, we sum the outputs of the two attention modules to further improve feature representations which contribute to more precise segmentation predictions across temporal and spatial dimensions simultaneously. Extensive experiments demonstrate the effectiveness of the proposed method, obtaining state-of-the-art mean intersection of union (mIoU) of 75.22% on CamVid dataset.

Introduction

Semantic segmentation—the task of assigning a category label to every pixel in an image—has seen massive gains thanks to Deep Convolutional Neural Networks (CNNs). However, most models are designed for static images. When applying these "image-centric" models to video, two major problems emerge:

  1. Temporal Inconsistency: Because the model looks at frames in isolation, the same object (like a car or a tree) might be classified inconsistently across adjacent frames, leading to distracting flickering.
  2. Spatial Inconsecutiveness: Standard classification backbones use repeated pooling and downsampling, which discards fine-grained spatial details. This often results in "broken" object predictions where parts of a single instance are misclassified.

The authors propose the Attention-Guided Network (AGNet) to exploit the strong temporal continuity and spatial context inherent in video sequences.

Methodology

AGNet sits on top of a dilated Fully Convolutional Network (FCN) backbone. The core innovation lies in two parallel attention modules designed to capture complementary information:

Architecture of the Adjacent Attention Module (AAM) and Spatial Attention Module (SAM).
  • Adjacent Attention Module (AAM): This module focuses on the temporal dimension. It takes feature maps from the current frame ($F_t$) and the previous frame ($F_{t-1}$) to calculate a correlation map. By identifying how features relate across adjacent frames, it selectively enhances features of the same object over time, ensuring the model "remembers" what it saw a fraction of a second ago.
  • Spatial Attention Module (SAM): This module focuses on the spatial dimension within a single frame. It uses a self-attention mechanism to model the relationship between any two pixels in a frame, regardless of their distance. This allows the model to use global context to fill in gaps and create more cohesive object shapes.

Finally, a Feature Fusion Module (FFM) combines the outputs of these two modules via element-wise summation before passing them to a final classifier.

Key Results

The model was evaluated on the CamVid dataset, achieving a state-of-the-art 75.22% mIoU.

Visual results demonstrate that AGNet is significantly better at identifying small or thin objects, such as traffic signs and pedestrians, which are often lost or blurred by other models. It also successfully maintains consistent labels for fences and sidewalks across multiple frames where baseline models fail.

Qualitative comparison between the baseline Dilated FCN and the proposed AGNet across five consecutive video frames.

Ablation Study Highlights

  • Adding the AAM to a ResNet-50 baseline improved the mIoU by 3.62%.
  • Adding the SAM individually improved it by 5.08%.
  • The combination of both modules confirmed their complementary nature, providing the highest accuracy.

Conclusion

AGNet effectively addresses the challenges of temporal flickering and fragmented predictions in video segmentation. By using attention mechanisms instead of computationally expensive optical flow, the network remains robust and efficient. Future work may involve exploring more powerful backbones and refined decoding strategies to further improve boundary precision.

Summary updated

Abstract

Remarkable success has been made by deep convolutional neural network (CNN) models in semantic image segmentation. However, most segmentation models are based on classification networks which tend to learn image-level features and lost abundant spatial information due to repeated pooling and downsampling operations, and the CNN-based methods are not robust to inputs, hence directly applying existing segmentation methods to semantic video segmentation will result in spatially inconsecutive and temporally inconsistent segmentation predictions within one instance and of the same objects across adjacent frames, respectively. To tackle this challenge, we propose an Attention-Guided Network (AGNet) to adaptively strengthen inter-frame and intra-frame features for more precise segmentation predictions. Specifically, we append an adjacent attention module (AAM) and a spatial attention module (SAM) on the top of dilated fully convolutional network (FCN), which model the feature correlations in temporal and spatial dimensions, respectively. The AAM selectively enhances the inter-frame features of the same objects across adjacent frames for temporally consistent predictions. Meanwhile, the SAM selectively aggregates the intra-frame features within one instance for spatially consecutive predictions. Finally, we sum the outputs of the two attention modules to further improve feature representations which contribute to more precise segmentation predictions across temporal and spatial dimensions simultaneously. Extensive experiments demonstrate the effectiveness of the proposed method, obtaining state-of-the-art mean intersection of union (mIoU) of 75.22% on CamVid dataset.
BibTeX
@inproceedings{li2019attention,
  author = {Jiangyun Li and Yikai Zhao and Jun Fu and Jiajia Wu and Jing Liu},
  title = {{Attention-Guided Network for Semantic Video Segmentation}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=xZW_ULm---},
}
Figure