An End-to-End Trainable Video Panoptic Segmentation Method usingTransformers

Jeongwon Ryu, Kwangjin YoonarXiv 2021 · arXiv

Introduction

Video Panoptic Segmentation (VPS) represents a significant challenge in computer vision, requiring the simultaneous semantic labeling of background "stuff" and the consistent tracking of individual "thing" instances across video frames. Traditionally, this task has been approached by processing frames independently and employing complex, heuristic-based post-processing to link object identities over time.

This paper introduces SIAin, a transformer-based framework designed to unify segmentation and tracking into a single, end-to-end trainable system. By extending the MaskFormer architecture, SIAin eliminates the need for disjointed tracking modules, leveraging the inherent temporal consistency of transformer embeddings to maintain object identities.

Methodology

SIAin builds upon the MaskFormer architecture, which treats segmentation as a mask classification problem. To adapt this for video, the authors introduced three primary innovations:

1. Query Propagation

The core of SIAin's tracking capability is Query Propagation. In a standard transformer decoder, queries are used to probe an image for objects. SIAin takes the object embeddings (mathematical representations of identity) generated in the previous frame and inserts them as queries into the decoder for the current frame. This allows the model to "remember" and directly refine the representation of specific objects across time.

The SIAin architecture propagates object embeddings as queries across frames to maintain identity.

2. Episode-based Training

To ensure the network learns temporal dynamics, it is trained on "episodes"—short sequences of 2 to 5 randomly sampled frames. This training regime forces the model to maintain consistent masks and identities throughout the sequence rather than optimizing for single-frame accuracy.

3. Dual Loss Function

A specialized loss function ($L$) was developed to balance two competing objectives:

  • Detection Loss ($L_{S,D}$): Focuses on correctly identifying semantic categories and discovering newly appearing objects in the scene.
  • Tracking Loss ($L_T$): Minimizes errors for objects that have already been identified in previous frames, ensuring stable identity propagation.

Experiments and Results

The model was evaluated on the STEP (Segmenting and Tracking Every Pixel) dataset, specifically the KITTI-STEP and MOTChallenge-STEP benchmarks. Performance was measured using the STQ (Segmentation and Tracking Quality) metric, which integrates spatial accuracy and temporal consistency into a single score.

Quantitative Performance

SIAin demonstrated a significant performance advantage over established baselines. On the KITTI-STEP benchmark, the model achieved an STQ of 57.87%, a substantial improvement over the Motion-DeepLab baseline of 52.19%.

SIAin outperforms the Motion-DeepLab baseline on the KITTI-STEP benchmark across all metrics.

On the more challenging, pedestrian-heavy MOTChallenge-STEP dataset, SIAin reached an STQ of 31.8%. Qualitative analysis confirms that the model maintains consistent color-coded IDs for vehicles and pedestrians even during complex maneuvers in urban environments.

Key Insights and Limitations

The primary insight of this work is that embeddings can serve as identity. By treating the transformer's internal state as a persistent "ID card," the model avoids the overhead of external re-identification modules. This unified approach maintains the efficiency and elegance of the transformer architecture while achieving state-of-the-art results.

However, limitations remain. The current system relies on a simple heuristic to terminate tracks (missing for 5 consecutive frames), which may fail during long-term occlusions. Furthermore, the computational intensity of the Swin-L backbone and multi-frame training suggests that further optimization is required for real-time or mobile applications.

Summary updated

BibTeX
@misc{ryu2021end,
  author = {Jeongwon Ryu and Kwangjin Yoon},
  title = {{An End-to-End Trainable Video Panoptic Segmentation Method usingTransformers}},
  year = {2021},
  eprint = {2110.04009},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.04009},
}
Figure