Introduction
Video Panoptic Segmentation (VPS) represents a significant challenge in computer vision, requiring the simultaneous semantic labeling of background "stuff" and the consistent tracking of individual "thing" instances across video frames. Traditionally, this task has been approached by processing frames independently and employing complex, heuristic-based post-processing to link object identities over time.
This paper introduces SIAin, a transformer-based framework designed to unify segmentation and tracking into a single, end-to-end trainable system. By extending the MaskFormer architecture, SIAin eliminates the need for disjointed tracking modules, leveraging the inherent temporal consistency of transformer embeddings to maintain object identities.
Methodology
SIAin builds upon the MaskFormer architecture, which treats segmentation as a mask classification problem. To adapt this for video, the authors introduced three primary innovations:
1. Query Propagation
The core of SIAin's tracking capability is Query Propagation. In a standard transformer decoder, queries are used to probe an image for objects. SIAin takes the object embeddings (mathematical representations of identity) generated in the previous frame and inserts them as queries into the decoder for the current frame. This allows the model to "remember" and directly refine the representation of specific objects across time.
2. Episode-based Training
To ensure the network learns temporal dynamics, it is trained on "episodes"—short sequences of 2 to 5 randomly sampled frames. This training regime forces the model to maintain consistent masks and identities throughout the sequence rather than optimizing for single-frame accuracy.
3. Dual Loss Function
A specialized loss function ($L$) was developed to balance two competing objectives:
- Detection Loss ($L_{S,D}$): Focuses on correctly identifying semantic categories and discovering newly appearing objects in the scene.
- Tracking Loss ($L_T$): Minimizes errors for objects that have already been identified in previous frames, ensuring stable identity propagation.
Experiments and Results
The model was evaluated on the STEP (Segmenting and Tracking Every Pixel) dataset, specifically the KITTI-STEP and MOTChallenge-STEP benchmarks. Performance was measured using the STQ (Segmentation and Tracking Quality) metric, which integrates spatial accuracy and temporal consistency into a single score.
Quantitative Performance
SIAin demonstrated a significant performance advantage over established baselines. On the KITTI-STEP benchmark, the model achieved an STQ of 57.87%, a substantial improvement over the Motion-DeepLab baseline of 52.19%.
On the more challenging, pedestrian-heavy MOTChallenge-STEP dataset, SIAin reached an STQ of 31.8%. Qualitative analysis confirms that the model maintains consistent color-coded IDs for vehicles and pedestrians even during complex maneuvers in urban environments.
Key Insights and Limitations
The primary insight of this work is that embeddings can serve as identity. By treating the transformer's internal state as a persistent "ID card," the model avoids the overhead of external re-identification modules. This unified approach maintains the efficiency and elegance of the transformer architecture while achieving state-of-the-art results.
However, limitations remain. The current system relies on a simple heuristic to terminate tracks (missing for 5 consecutive frames), which may fail during long-term occlusions. Furthermore, the computational intensity of the Swin-L backbone and multi-frame training suggests that further optimization is required for real-time or mobile applications.
Summary updated