Large-scale Video Panoptic Segmentation in the Wild: A Benchmark

Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, Yi Yang

Introduction

Video Panoptic Segmentation (VPS) represents a significant challenge in computer vision, requiring models to assign both a semantic label and a unique tracking ID to every pixel in a video sequence. While tasks like image panoptic segmentation have matured, VPS has been bottlenecked by the lack of diverse, large-scale data. Existing benchmarks like Cityscapes-VPS and KITTI-STEP are primarily restricted to driving scenes, failing to capture the complexity of "in the wild" environments.

To address this, we introduce VIPSeg, the first large-scale dataset for video panoptic segmentation in diverse real-world scenarios. Alongside this benchmark, we propose Clip-PanoFCN, a clip-based baseline that improves upon traditional frame-by-frame iterative processing by enhancing both computational efficiency and temporal consistency.

The VIPSeg Dataset

VIPSeg is designed to push the boundaries of VPS beyond the street-view domain. It comprises 3,536 videos and 84,750 frames, covering 232 distinct scenarios—ranging from kindergartens to natural grottos.

Visual examples of the VIPSeg dataset, showcasing panoptic masks across a wide variety of real-world scenes.

The dataset includes 124 semantic categories, divided into 58 "thing" classes (objects with distinct identities) and 66 "stuff" classes (amorphous regions like sky or grass). In terms of scale, VIPSeg is approximately 6x larger than Cityscapes-VPS and 60x larger than KITTI-STEP by video count, providing the necessary variety to train robust models for real-world applications.

Methodology

Sparse-to-Dense Interactive Annotation

Annotating nearly 85,000 frames at the pixel level is a massive undertaking. The authors developed a semi-automated pipeline to facilitate this process:

  1. Sparse Phase: Human annotators label frames at a low frequency (1 fps).
  2. Dense Phase: A Video Object Segmentation (VOS) model propagates these masks to a higher frame rate (5 fps).
  3. Refinement: Experts manually correct machine-generated masks, particularly in dense or complex scenes.

Clip-PanoFCN Baseline

Most existing VPS models process videos iteratively, which is prone to "drift"—where tracking IDs are lost or swapped during occlusions. Clip-PanoFCN adopts a clip-based approach to mitigate this:

  • Frame-level Modeling: The model generates object "kernels" and feature maps for each frame.
  • Clip-level Aggregation: Within a short temporal window (e.g., 8 frames), kernels representing the same object are fused via average clustering to create a stable identity.
  • Inter-clip Tracking: Identities are maintained across the full video by matching fused kernels between adjacent clips using cosine similarity.

Key Insights and Results

Evaluations highlight the increased difficulty of "in the wild" segmentation compared to constrained driving datasets. Clip-PanoFCN achieved a Video Panoptic Quality (VPQ) of 22.9 and a Segmentation and Tracking Quality (STQ) of 31.5 on VIPSeg, outperforming previous methods like VPSNet and VIP-DeepLab.

The distribution of instances across the 124 categories in VIPSeg, illustrating the long-tail nature of real-world data.

A significant challenge identified is the "long-tail" distribution of classes. While common categories like "person" and "car" are abundant, rare classes such as "commode" or "fishbowl" remain difficult for models to generalize. The study also revealed a trade-off regarding clip length: while longer clips improve temporal stability (STQ) by providing more context, they can degrade raw segmentation accuracy (VPQ) if an object's appearance changes too drastically within the window.

Conclusion

VIPSeg provides a much-needed benchmark for evaluating VPS models in complex, real-world environments. By moving from frame-based to clip-based processing with Clip-PanoFCN, the authors demonstrate improved temporal stability and efficiency. Future research will need to address the challenges of long-tailed distributions and the high human cost of high-quality video annotation.

Summary updated

Abstract

In this paper, we present a new large-scale dataset for the video panoptic segmentation task, which aims to assign semantic classes and track identities to all pixels in a video. As the ground truth for this task is difficult to annotate, previous datasets for video panoptic segmentation are limited by either small scales or the number of scenes. In contrast, our large-scale VIdeo Panoptic Segmentation in the Wild (VIPSeg) dataset provides 3,536 videos and 84,750 frames with pixel-level panoptic annotations, covering a wide range of real-world scenarios and categories. To the best of our knowledge, our VIPSeg is the first attempt to tackle the challenging video panoptic segmentation task in the wild by considering diverse scenarios. Based on VIPSeg, we evaluate existing video panoptic segmentation approaches and propose an efficient and effective clip-based baseline method to analyze our VIPSeg dataset. Our dataset is available at https://github.com/VIPSeg-Dataset/VIPSeg-Dataset/.
BibTeX
@inproceedings{miao2022large,
  author = {Jiaxu Miao and Xiaohan Wang and Yu Wu and Wei Li and Xu Zhang and Yunchao Wei and Yi Yang},
  title = {{Large-scale Video Panoptic Segmentation in the Wild: A Benchmark}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.02036},
}
Figure