TL;DR
TubeFormer-DeepLab is a unified video segmentation framework that treats semantic, instance, and panoptic segmentation as a single task: predicting class-labeled "video tubes." By leveraging a hierarchical dual-path transformer with latent and global memories, it achieves state-of-the-art performance across multiple benchmarks with a single, streamlined architecture.
What Problem Does This Paper Address?
In the computer vision community, video semantic (VSS), instance (VIS), and panoptic (VPS) segmentation have historically been treated as distinct problems. This has led to a "bifurcation" of research: VSS models often rely on feature warping between frames, VIS models focus on detection-and-tracking, and VPS models use complex, multi-headed pipelines to combine the two.
The authors of TubeFormer-DeepLab argue that this divergence is unnecessary. They observe that all these tasks share a common goal: partitioning a video into spatio-temporal "tubes"—sequences of masks linked over time—and assigning them labels. Whether those labels represent a semantic category (VSS), an instance ID (VIS), or both (VPS) is just a matter of task configuration.
Methods
The core of TubeFormer-DeepLab is a Mask Transformer adapted for the video domain. Instead of classifying every pixel, the model predicts a set of $N$ pairs, each consisting of a class prediction and a "tube embedding" vector. This embedding is then multiplied by high-resolution pixel features to generate the final video tube.
To handle the computational complexity of video data, the authors introduce a Hierarchical Dual-Path Transformer:
- Latent Dual-Path Transformer: This block processes individual frames in parallel. It uses a small, learnable "latent memory" to pass messages between frame-level features, effectively distilling spatial information.
- Global Dual-Path Transformer: This block operates on the entire video clip. It uses a "global memory" to learn spatio-temporal relationships, allowing the model to cluster pixels into tubes across time.
Additional innovations include:
- Split Global Memory: The global memory is divided into "thing-specific" and "stuff-specific" sets to better handle the different natures of countable objects versus amorphous background regions.
- Temporal Consistency Loss: A loss function that encourages the model to produce consistent predictions in the overlapping frames between video clips.
- Clip-Paste: A data augmentation strategy that randomly pastes entire video tubes (objects or background) from one clip into another.
Datasets
The model was evaluated on five major benchmarks covering different segmentation tasks:
- KITTI-STEP: Video panoptic segmentation (VPS).
- VIPSeg: A recent, large-scale dataset for diverse in-the-wild VPS.
- VSPW: A large-scale benchmark for video semantic segmentation (VSS).
- YouTube-VIS (2019 & 2021): The standard benchmarks for video instance segmentation (VIS).
- SemKITTI-DVPS: A dataset for depth-aware video panoptic segmentation.
Key Insights and Ideas
- Unification is Simplification: By formulating video segmentation as tube prediction, the authors eliminate the need for task-specific heads (like tracking branches or warping modules), resulting in a much simpler end-to-end system.
- Hierarchical Attention is Essential: Directly attending to every pixel in a video clip is too noisy and computationally expensive. The two-stage hierarchy (latent then global) acts as a powerful feature aggregator.
- Memory Specialization: Visualizations show that the latent memory learns to specialize in specific spatial areas or boundaries, while the global memory learns to focus on individual object tubes.
Key Results
TubeFormer-DeepLab set new state-of-the-art records across nearly every benchmark it touched:
- VPS: On KITTI-STEP, it outperformed the previous best published method (Motion-DeepLab) by +13.1 STQ.
- VSS: On the VSPW test set, it achieved a massive +21 mIoU improvement over the TCB baseline.
- VIS: On YouTube-VIS 2019, it outperformed other end-to-end transformers like VisTR by +7.4 AP and IFC by +2.9 AP.
- DVPS: It achieved a new state-of-the-art of 67.0 DSTQ on SemKITTI-DVPS, surpassing ViP-DeepLab by +3.4.
Limitations and Future Work
The primary limitation is that the model currently operates on relatively short video clips ($T=2$ to $5$ frames). While it uses a stitching heuristic to handle long videos, it performs "short-term tracking" and can occasionally lose track of objects that disappear and reappear over long durations. The authors suggest that efficiently incorporating long-term tracking directly into the transformer architecture is a promising direction for future research.
Summary updated