PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation

Shilin Yan, Xiaohao Xu, Renrui Zhang, Lingyi Hong, Wenchao Chen, Wenqiang Zhang, Wei Zhang

TL;DR

PanoVOS introduces the first large-scale dataset specifically designed for panoramic video object segmentation, addressing the unique challenges of 360° content. The authors also propose PSCFormer, a novel Transformer-based architecture that uses a specialized "stitching" attention mechanism to maintain spatial consistency across the artificial boundaries of panoramic images.

What Problem Does This Paper Address?

Video Object Segmentation (VOS) has seen incredible progress, but most research is confined to the "flat" world of conventional 2D planar images. However, panoramic (360°) video is becoming essential for fields like autonomous driving and virtual reality.

When you project a 360° sphere onto a flat 2D plane (like an equirectangular projection), two major problems arise that break standard VOS models:

  1. Content Discontinuity: An object moving off the right edge of the image immediately reappears on the left edge. Standard models treat these as two separate, disconnected regions.
  2. Severe Distortion: Objects near the top or bottom of the frame undergo extreme geometric warping, making them difficult for models trained on "normal" images to recognize.

The authors found that even state-of-the-art (SOTA) models suffer a massive performance drop (often over 20% in accuracy) when moved from planar to panoramic data.

Fig. 1: Panoramic video object segmentation (PanoVOS). PanoVOS targets tracking and distinguishing the particular instances under content discontinuities (e.g. penguin in the image of T = 15) and serve distortion (e.g. penguin in the image of T = 65). We show the sample of (a) frames, (b) segmentation annotations, and (c) area proportion of foreground for the Penguin video in our dataset.

Methods

To bridge this gap, the authors developed PSCFormer (Panoramic Space Consistency Transformer). The core innovation is the Panoramic Space Consistency (PSC) block.

Standard attention mechanisms in Transformers look at relationships between pixels within a frame. PSCFormer goes a step further with PSC-Attention:

  • Boundary Stitching: It mathematically "stitches" the left and right boundaries in the feature space. By shifting a portion of the feature map from one side to the other during the attention calculation, the model learns that these edges are actually continuous.
  • Fixed Window Matching: To keep computation manageable, the attention focuses on a fixed window size, but one that is aware of the panoramic wrap-around.
  • Temporal Consistency: It utilizes a memory encoder to match the current "query" frame against multiple reference frames (the first frame and previous frames), ensuring the object is tracked accurately over time even if it disappears and reappears across the seam.
Fig. 5: The PSCFormer architecture and the details of the Panoramic Space Consistency (PSC) block.

Datasets

The paper introduces PanoVOS, a high-quality benchmark for this new task:

  • Scale: 150 videos with 19,145 manually refined instance masks across 35 categories (people, animals, and common objects).
  • Quality: Nearly half of the videos are in 4K resolution.
  • Complexity: The average video length is 20 seconds—roughly four times longer than standard VOS datasets like DAVIS—introducing more opportunities for complex motion and object re-entry.

The authors used a semi-automatic annotation pipeline, where experts annotated keyframes at 1 fps, a SOTA model (AOT) propagated those masks, and humans manually refined the results to correct for panoramic errors.

Fig. 3: Instance-level distribution of PanoVOS dataset. Our dataset contains three major divisions: person, animals, and common objects with 35 sub-divisions.

Key Insights and Ideas

  • The "Seam" is the Enemy: The artificial boundary in equirectangular projections is the primary cause of failure for existing trackers. PSCFormer’s ability to "see through" this seam is its greatest strength.
  • Domain Gap is Quantifiable: By testing 15 off-the-shelf models, the authors proved that simply training on more data (like the massive YouTube-VOS) helps, but it doesn't solve the fundamental geometric mismatch of panoramic views.
  • Efficiency Matters: Directly splicing images to handle boundaries would double the computation. The PSC-Attention approach achieves the same goal within the feature space much more efficiently.

Key Results

PSCFormer significantly outperforms existing SOTA models when evaluated on the PanoVOS benchmark:

  • Quantitative: The "Large" version of PSCFormer achieved a 77.9 J&F score on the validation set, outperforming the previous best model, AOTL (66.6), by a wide margin.
  • Ablation: Removing the PSC-Attention module caused a drop of about 4% in J&F, proving that the panoramic-aware design is doing the heavy lifting.
  • Generalization: The model showed strong performance even on "unseen" categories in the test set, suggesting it learns robust features rather than just memorizing shapes.

Limitations and Future Work

While PSCFormer handles boundary discontinuities well, the authors acknowledge it still struggles with extreme geometric distortion (e.g., objects stretched thin near the poles). They suggest that incorporating specialized designs like deformable convolutions could be a promising next step.

Additionally, the authors hope PanoVOS will serve as a foundation for broader panoramic research, including tasks like referring video object segmentation (finding objects based on text descriptions) and embodied navigation.

Summary updated

Abstract

Panoramic videos contain richer spatial information and have attracted tremendous amounts of attention due to their exceptional experience in some fields such as autonomous driving and virtual reality. However, existing datasets for video segmentation only focus on conventional planar images. To address the challenge, in this paper, we present a panoramic video dataset, PanoVOS. The dataset provides 150 videos with high video resolutions and diverse motions. To quantify the domain gap between 2D planar videos and panoramic videos, we evaluate 15 off-the-shelf video object segmentation (VOS) models on PanoVOS. Through error analysis, we found that all of them fail to tackle pixel-level content discontinues of panoramic videos. Thus, we present a Panoramic Space Consistency Transformer (PSCFormer), which can effectively utilize the semantic boundary information of the previous frame for pixel-level matching with the current frame. Extensive experiments demonstrate that compared with the previous SOTA models, our PSCFormer network exhibits a great advantage in terms of segmentation results under the panoramic setting. Our dataset poses new challenges in panoramic VOS and we hope that our PanoVOS can advance the development of panoramic segmentation/tracking.
BibTeX
@inproceedings{yan2024panovos,
  author = {Shilin Yan and Xiaohao Xu and Renrui Zhang and Lingyi Hong and Wenchao Chen and Wenqiang Zhang and Wei Zhang},
  title = {{PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72673-6\_19},
}
Figure