Abstract
One major challenge for semantic segmentation in real-world scenarios is the limited availability of pixel-level labels due to the high expense of human labor, even when vast volumes of video data are available. Existing semi-supervised methods attempt to exploit unlabeled data in model training but often treat video as a set of independent images. To better explore the semi-supervised segmentation problem with video data, we formulate a semi-supervised video semantic segmentation task. We observe that overfitting is surprisingly severe between labeled and unlabeled frames within a training video, despite their visual similarity. This phenomenon, termed "inner-video overfitting," leads to inferior performance. To tackle this, we propose a novel inter-frame feature reconstruction (IFR) technique to leverage ground-truth labels to supervise model training on unlabeled frames. IFR utilizes the internal relevance of different frames within a video to narrow the feature distributions between labeled and unlabeled frames. Extensive experiments on Cityscapes and CamVid demonstrate the superiority of our proposed method over previous state-of-the-art methods.
Introduction
Semantic segmentation—the task of assigning a category to every pixel in an image—is a cornerstone of computer vision, yet it remains notoriously data-hungry. While video data is plentiful, labeling it is incredibly expensive; for instance, the popular Cityscapes dataset only provides annotations for one out of every 20 frames.
Existing Semi-Supervised Image Segmentation (SSIS) methods fail to fully exploit the temporal nature of video because they treat frames as independent entities. This paper identifies a specific challenge called inner-video overfitting: models trained on a few labeled frames often fail to generalize to the unlabeled frames of that same video, despite the frames being visually similar in style and content.
Methodology: Inter-Frame Feature Reconstruction (IFR)
To address inner-video overfitting, the authors introduce the Inter-Frame Feature Reconstruction (IFR) technique. Instead of relying solely on standard consistency losses (which compare final predictions), IFR uses the internal relevance of video frames to create a stronger supervision signal at the feature level.
The IFR process follows four key steps:
- Feature Extraction: The model extracts features from a labeled frame ($x_{t1}$) and an unlabeled frame ($x_{t2}$) from the same video.
- Prototype Generation: It calculates "class prototypes" (feature centroids for each category) from the unlabeled frame $x_{t2}$ using pseudo-labels.
- Reconstruction via Attention: Using an attention mechanism, the model attempts to reconstruct the features of the labeled frame ($x_{t1}$) by combining the prototypes derived from the unlabeled frame ($x_{t2}$).
- Supervised Reconstruction Loss: Because the ground-truth label for $x_{t1}$ is available, the model can be supervised on this reconstructed feature map.
This process forces the feature distributions of labeled and unlabeled frames to align. If the model can successfully reconstruct a labeled frame's features using an unlabeled frame's components, it demonstrates a robust representation that transcends specific frames.
Experimental Results
The researchers evaluated IFR on two primary benchmarks, Cityscapes and CamVid, using various "partition protocols" (different amounts of labeled data).
Performance Gains
IFR achieved state-of-the-art (SOTA) results across the board:
- Cityscapes (1/30 labels): Reached 52.86% mIoU, a significant +7.13% improvement over the supervised baseline and nearly 4% higher than the previous best semi-supervised method (CPS).
- CamVid (1/30 labels): Achieved 49.50% mIoU, also marking a +7.13% gain over the baseline.
Temporal Consistency
Beyond accuracy, the method improved the temporal stability of the video (TC score). This means the resulting segmentation masks exhibit less "flicker" between frames compared to standard image-based methods, which is critical for real-world applications like autonomous driving.
Conclusion
By identifying and addressing the "inner-video overfitting" paradox, this work demonstrates that the actual ground truth of one frame can be used to supervise the features of another frame via a reconstruction bridge. This approach discourages the model from learning frame-specific shortcuts and encourages the learning of underlying scene semantics. While future work may integrate IFR with more complex architectures utilizing long-range temporal dependencies, the current results set a new benchmark for semi-supervised video segmentation.
Summary updated