Semi-Supervised Video Semantic Segmentation with Inter-Frame Feature Reconstruction

Jiafan Zhuang, Zilei Wang, Yuan GaoCVPR 2022 · OpenReview

Abstract

One major challenge for semantic segmentation in real-world scenarios is the limited availability of pixel-level labels due to the high expense of human labor, even when vast volumes of video data are available. Existing semi-supervised methods attempt to exploit unlabeled data in model training but often treat video as a set of independent images. To better explore the semi-supervised segmentation problem with video data, we formulate a semi-supervised video semantic segmentation task. We observe that overfitting is surprisingly severe between labeled and unlabeled frames within a training video, despite their visual similarity. This phenomenon, termed "inner-video overfitting," leads to inferior performance. To tackle this, we propose a novel inter-frame feature reconstruction (IFR) technique to leverage ground-truth labels to supervise model training on unlabeled frames. IFR utilizes the internal relevance of different frames within a video to narrow the feature distributions between labeled and unlabeled frames. Extensive experiments on Cityscapes and CamVid demonstrate the superiority of our proposed method over previous state-of-the-art methods.

Introduction

Semantic segmentation—the task of assigning a category to every pixel in an image—is a cornerstone of computer vision, yet it remains notoriously data-hungry. While video data is plentiful, labeling it is incredibly expensive; for instance, the popular Cityscapes dataset only provides annotations for one out of every 20 frames.

Existing Semi-Supervised Image Segmentation (SSIS) methods fail to fully exploit the temporal nature of video because they treat frames as independent entities. This paper identifies a specific challenge called inner-video overfitting: models trained on a few labeled frames often fail to generalize to the unlabeled frames of that same video, despite the frames being visually similar in style and content.

Methodology: Inter-Frame Feature Reconstruction (IFR)

To address inner-video overfitting, the authors introduce the Inter-Frame Feature Reconstruction (IFR) technique. Instead of relying solely on standard consistency losses (which compare final predictions), IFR uses the internal relevance of video frames to create a stronger supervision signal at the feature level.

The training data setup for semi-supervised video semantic segmentation, illustrating the relationship between sampled labeled frames and the remaining unlabeled frames within a video sequence.

The IFR process follows four key steps:

  1. Feature Extraction: The model extracts features from a labeled frame ($x_{t1}$) and an unlabeled frame ($x_{t2}$) from the same video.
  2. Prototype Generation: It calculates "class prototypes" (feature centroids for each category) from the unlabeled frame $x_{t2}$ using pseudo-labels.
  3. Reconstruction via Attention: Using an attention mechanism, the model attempts to reconstruct the features of the labeled frame ($x_{t1}$) by combining the prototypes derived from the unlabeled frame ($x_{t2}$).
  4. Supervised Reconstruction Loss: Because the ground-truth label for $x_{t1}$ is available, the model can be supervised on this reconstructed feature map.

This process forces the feature distributions of labeled and unlabeled frames to align. If the model can successfully reconstruct a labeled frame's features using an unlabeled frame's components, it demonstrates a robust representation that transcends specific frames.

Experimental Results

The researchers evaluated IFR on two primary benchmarks, Cityscapes and CamVid, using various "partition protocols" (different amounts of labeled data).

Performance Gains

IFR achieved state-of-the-art (SOTA) results across the board:

  • Cityscapes (1/30 labels): Reached 52.86% mIoU, a significant +7.13% improvement over the supervised baseline and nearly 4% higher than the previous best semi-supervised method (CPS).
  • CamVid (1/30 labels): Achieved 49.50% mIoU, also marking a +7.13% gain over the baseline.

Temporal Consistency

Beyond accuracy, the method improved the temporal stability of the video (TC score). This means the resulting segmentation masks exhibit less "flicker" between frames compared to standard image-based methods, which is critical for real-world applications like autonomous driving.

Conclusion

By identifying and addressing the "inner-video overfitting" paradox, this work demonstrates that the actual ground truth of one frame can be used to supervise the features of another frame via a reconstruction bridge. This approach discourages the model from learning frame-specific shortcuts and encourages the learning of underlying scene semantics. While future work may integrate IFR with more complex architectures utilizing long-range temporal dependencies, the current results set a new benchmark for semi-supervised video segmentation.

Summary updated

Abstract

One major challenge for semantic segmentation in realworld scenarios is only limited pixel-level labels available due to high expense of human labor though a vast volume of video data is provided. Existing semi-supervised methods attempt to exploit unlabeled data in model training, but they just regard video as a set of independent images. To better explore semi-supervised segmentation problem with video data, we formulate a semi-supervised video semantic segmentation task in this paper. For this task, we observe that the overfitting is surprisingly severe between labeled and unlabeled frames within a training video although they are very similar in style and contents. This is called inner-video overfitting, and it would actually lead to inferior performance. To tackle this issue, we propose a novel interframe feature reconstruction (IFR) technique to leverage the ground-truth labels to supervise the model training on unlabeled frames. IFR is essentially to utilize the internal relevance of different frames within a video. During training, IFR would enforce the feature distributions between labeled and unlabeled frames to be narrowed. Consequently, the inner-video overfitting issue can be effectively alleviated. We conduct extensive experiments on Cityscapes and CamVid, and the results demonstrate the superiority of our proposed method to previous state-of-the-art methods. The code is available at https://github.com/jfzhuang/IFR.
BibTeX
@inproceedings{zhuang2022semi,
  author = {Jiafan Zhuang and Zilei Wang and Yuan Gao},
  title = {{Semi-Supervised Video Semantic Segmentation with Inter-Frame Feature Reconstruction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=59kdgYRbsC},
}
Figure