Simultaneously Short- and Long-Term Temporal Modeling for Semi-Supervised Video Semantic Segmentation

Jiangwei Lao, Weixiang Hong, Xin Guo, Yingying Zhang, Jian Wang, Jingdong Chen, Wei ChuCVPR 2023 · DOI

Abstract

Training video semantic segmentation (VSS) models is notoriously expensive due to the high cost of dense, frame-by-frame labeling. This paper introduces SSLTM (Simultaneously Short- and Long-Term Temporal Modeling), a framework designed to achieve high-fidelity segmentation using only one labeled frame per video. By simultaneously modeling short-term motion through adjacent frames and long-term context through distant reference frames, SSLTM achieves performance levels that rival fully supervised models trained on dense annotations.

The Challenge of Video Annotation

Semantic segmentation—assigning a category to every pixel—is a fundamental task in computer vision. In video, this task is complicated by motion blur, occlusions, and temporal inconsistencies. While deep learning has excelled in this domain, it typically requires massive amounts of labeled data. "Dense annotation" is labor-intensive and often redundant, as adjacent frames share significant information. The authors address the extreme semi-supervised setting: training a model where only the first frame of each video is labeled. Existing methods often overfit to these sparse labels or fail to capture the broader context provided by the rest of the unlabeled video.

The SSLTM Framework

To overcome these limitations, the authors propose SSLTM, which integrates three core components to handle different temporal scales:

  1. Spatial-Temporal Transformer (STT): This component manages "short-term" dependencies. It employs a 3D window-based attention mechanism to fuse the current frame with its immediate neighbors, ensuring local motion consistency and capturing fine-grained temporal details.
  2. Reference Frame Context Enhancement (RFCE): This module addresses "long-term" context. During training, the model samples a distant, unlabeled "reference" frame from the same video. Using an Object-Contextual Representation (OCR) module, it extracts category-level features to "remind" the current frame of object appearances, which is particularly useful when objects are occluded or blurred in the current view.
  3. Global Category Context (GCC): To handle cases where a category in the current frame is missing from the sampled reference frame, GCC maintains a learnable global memory of all categories across the dataset, filling in contextual gaps.
The SSLTM architecture integrates short-term temporal representations with long-term reference frame context and global category knowledge.

Key Insights: Distant Frames as Regularizers

A primary insight of this work is that incorporating distant unlabeled frames during training acts as a powerful regularizer. It prevents the model from overfitting to the few labeled frames by forcing the feature extractor to learn representations that remain robust across the entire video sequence. Furthermore, the model utilizes category-level representations rather than pixel-to-pixel matching for long-term modeling. This approach is more computationally efficient and less susceptible to noise, allowing the model to effectively transfer knowledge from distant frames to the current query frame.

Qualitative comparison showing how the inclusion of a distant reference frame (RFCE) corrects segmentation errors in the current frame, such as the car in the red box.

Experimental Results

The model was evaluated on the VSPW (Video Scene Parsing in the Wild) and Cityscapes benchmarks. Key findings include:

  • State-of-the-Art Performance: In the single-frame-labeled setting on VSPW, SSLTM outperformed previous methods by 2% to 3% mIoU.
  • Closing the Supervision Gap: When paired with pseudo-labeling (MeanTeacher), SSLTM reached 40.49% mIoU, coming within 0.13% of the performance ceiling established by models trained on every single frame manually.
  • Temporal Stability: Beyond accuracy, SSLTM demonstrated superior temporal consistency, significantly reducing the "flickering" artifacts common in image-based segmentation models.

Conclusion

SSLTM demonstrates that the high redundancy in video data can be leveraged to drastically reduce annotation costs. By balancing local motion and global context, the framework achieves near-perfect performance with minimal supervision. Future work may explore identifying the most informative frames for labeling to further enhance efficiency in extremely long, untrimmed video sequences.

Summary updated

BibTeX
@inproceedings{lao2023simultaneously,
  author = {Jiangwei Lao and Weixiang Hong and Xin Guo and Yingying Zhang and Jian Wang and Jingdong Chen and Wei Chu},
  title = {{Simultaneously Short- and Long-Term Temporal Modeling for Semi-Supervised Video Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01418},
}
Figure