Introduction
Video Panoptic Segmentation (VPS) represents a significant challenge in computer vision, requiring models to assign both a semantic label and a unique tracking ID to every pixel in a video sequence. While tasks like image panoptic segmentation have matured, VPS has been bottlenecked by the lack of diverse, large-scale data. Existing benchmarks like Cityscapes-VPS and KITTI-STEP are primarily restricted to driving scenes, failing to capture the complexity of "in the wild" environments.
To address this, we introduce VIPSeg, the first large-scale dataset for video panoptic segmentation in diverse real-world scenarios. Alongside this benchmark, we propose Clip-PanoFCN, a clip-based baseline that improves upon traditional frame-by-frame iterative processing by enhancing both computational efficiency and temporal consistency.
The VIPSeg Dataset
VIPSeg is designed to push the boundaries of VPS beyond the street-view domain. It comprises 3,536 videos and 84,750 frames, covering 232 distinct scenarios—ranging from kindergartens to natural grottos.
The dataset includes 124 semantic categories, divided into 58 "thing" classes (objects with distinct identities) and 66 "stuff" classes (amorphous regions like sky or grass). In terms of scale, VIPSeg is approximately 6x larger than Cityscapes-VPS and 60x larger than KITTI-STEP by video count, providing the necessary variety to train robust models for real-world applications.
Methodology
Sparse-to-Dense Interactive Annotation
Annotating nearly 85,000 frames at the pixel level is a massive undertaking. The authors developed a semi-automated pipeline to facilitate this process:
- Sparse Phase: Human annotators label frames at a low frequency (1 fps).
- Dense Phase: A Video Object Segmentation (VOS) model propagates these masks to a higher frame rate (5 fps).
- Refinement: Experts manually correct machine-generated masks, particularly in dense or complex scenes.
Clip-PanoFCN Baseline
Most existing VPS models process videos iteratively, which is prone to "drift"—where tracking IDs are lost or swapped during occlusions. Clip-PanoFCN adopts a clip-based approach to mitigate this:
- Frame-level Modeling: The model generates object "kernels" and feature maps for each frame.
- Clip-level Aggregation: Within a short temporal window (e.g., 8 frames), kernels representing the same object are fused via average clustering to create a stable identity.
- Inter-clip Tracking: Identities are maintained across the full video by matching fused kernels between adjacent clips using cosine similarity.
Key Insights and Results
Evaluations highlight the increased difficulty of "in the wild" segmentation compared to constrained driving datasets. Clip-PanoFCN achieved a Video Panoptic Quality (VPQ) of 22.9 and a Segmentation and Tracking Quality (STQ) of 31.5 on VIPSeg, outperforming previous methods like VPSNet and VIP-DeepLab.
A significant challenge identified is the "long-tail" distribution of classes. While common categories like "person" and "car" are abundant, rare classes such as "commode" or "fishbowl" remain difficult for models to generalize. The study also revealed a trade-off regarding clip length: while longer clips improve temporal stability (STQ) by providing more context, they can degrade raw segmentation accuracy (VPQ) if an object's appearance changes too drastically within the window.
Conclusion
VIPSeg provides a much-needed benchmark for evaluating VPS models in complex, real-world environments. By moving from frame-based to clip-based processing with Clip-PanoFCN, the authors demonstrate improved temporal stability and efficiency. Future research will need to address the challenges of long-tailed distributions and the high human cost of high-quality video annotation.
Summary updated