TubeFormer-DeepLab: Video Mask Transformer

Dahun Kim, Jun Xie, Huiyu Wang, Siyuan Qiao, Qihang Yu, Hong-Seok Kim, Hartwig Adam, In So Kweon, Liang-Chieh Chen

TL;DR

TubeFormer-DeepLab is a unified video segmentation framework that treats semantic, instance, and panoptic segmentation as a single task: predicting class-labeled "video tubes." By leveraging a hierarchical dual-path transformer with latent and global memories, it achieves state-of-the-art performance across multiple benchmarks with a single, streamlined architecture.

What Problem Does This Paper Address?

In the computer vision community, video semantic (VSS), instance (VIS), and panoptic (VPS) segmentation have historically been treated as distinct problems. This has led to a "bifurcation" of research: VSS models often rely on feature warping between frames, VIS models focus on detection-and-tracking, and VPS models use complex, multi-headed pipelines to combine the two.

The authors of TubeFormer-DeepLab argue that this divergence is unnecessary. They observe that all these tasks share a common goal: partitioning a video into spatio-temporal "tubes"—sequences of masks linked over time—and assigning them labels. Whether those labels represent a semantic category (VSS), an instance ID (VIS), or both (VPS) is just a matter of task configuration.

Figure 1. Video segmentation tasks can be formulated as partition- ing video frames (e.g., a clip) into tubes (i.e., segmentation masks linked along time) with different labels. TubeFormer-DeepLab di- rectly predicts class-labeled tubes, providing a simple and general solution to Video Semantic Segmentation (VSS), Video Instance Segmentation (VIS), and Video Panoptic Segmentation (VPS).

Methods

The core of TubeFormer-DeepLab is a Mask Transformer adapted for the video domain. Instead of classifying every pixel, the model predicts a set of $N$ pairs, each consisting of a class prediction and a "tube embedding" vector. This embedding is then multiplied by high-resolution pixel features to generate the final video tube.

To handle the computational complexity of video data, the authors introduce a Hierarchical Dual-Path Transformer:

  1. Latent Dual-Path Transformer: This block processes individual frames in parallel. It uses a small, learnable "latent memory" to pass messages between frame-level features, effectively distilling spatial information.
  2. Global Dual-Path Transformer: This block operates on the entire video clip. It uses a "global memory" to learn spatio-temporal relationships, allowing the model to cluster pixels into tubes across time.
Figure 3. TubeFormer-DeepLab architecture overview. TubeFormer-DeepLab extends the mask transformer [75] to generate a set of pairs, each containing a class prediction p(c) and a tube embedding vector w. The tube embedding vector, multiplied by the video pixel embedding features xv′ obtained by a convolutional network, yields the tube prediction ˆm. We introduce a hierarchical structure with the latent dual-path transformer block that is in charge of passing messages between frame-level features xf and a latent memory xl, followed by the global dual-path transformer block that learns the attentions between video-clip features xv and a global memory xm.

Additional innovations include:

  • Split Global Memory: The global memory is divided into "thing-specific" and "stuff-specific" sets to better handle the different natures of countable objects versus amorphous background regions.
  • Temporal Consistency Loss: A loss function that encourages the model to produce consistent predictions in the overlapping frames between video clips.
  • Clip-Paste: A data augmentation strategy that randomly pastes entire video tubes (objects or background) from one clip into another.

Datasets

The model was evaluated on five major benchmarks covering different segmentation tasks:

  • KITTI-STEP: Video panoptic segmentation (VPS).
  • VIPSeg: A recent, large-scale dataset for diverse in-the-wild VPS.
  • VSPW: A large-scale benchmark for video semantic segmentation (VSS).
  • YouTube-VIS (2019 & 2021): The standard benchmarks for video instance segmentation (VIS).
  • SemKITTI-DVPS: A dataset for depth-aware video panoptic segmentation.

Key Insights and Ideas

  • Unification is Simplification: By formulating video segmentation as tube prediction, the authors eliminate the need for task-specific heads (like tracking branches or warping modules), resulting in a much simpler end-to-end system.
  • Hierarchical Attention is Essential: Directly attending to every pixel in a video clip is too noisy and computationally expensive. The two-stage hierarchy (latent then global) acts as a powerful feature aggregator.
  • Memory Specialization: Visualizations show that the latent memory learns to specialize in specific spatial areas or boundaries, while the global memory learns to focus on individual object tubes.
Figure 2. Our proposed hierarchical dual-path transformer per- forms attention on three consecutive input frames (a) for VSS, VIS, and VPS tasks. While the global memory learns the spatio- temporally clustered attention for individual tube regions (b), our latent memory learns task-specific attention (c).

Key Results

TubeFormer-DeepLab set new state-of-the-art records across nearly every benchmark it touched:

  • VPS: On KITTI-STEP, it outperformed the previous best published method (Motion-DeepLab) by +13.1 STQ.
  • VSS: On the VSPW test set, it achieved a massive +21 mIoU improvement over the TCB baseline.
  • VIS: On YouTube-VIS 2019, it outperformed other end-to-end transformers like VisTR by +7.4 AP and IFC by +2.9 AP.
  • DVPS: It achieved a new state-of-the-art of 67.0 DSTQ on SemKITTI-DVPS, surpassing ViP-DeepLab by +3.4.

Limitations and Future Work

The primary limitation is that the model currently operates on relatively short video clips ($T=2$ to $5$ frames). While it uses a stitching heuristic to handle long videos, it performs "short-term tracking" and can occasionally lose track of objects that disappear and reappear over long durations. The authors suggest that efficiently incorporating long-term tracking directly into the transformer architecture is a promising direction for future research.

Summary updated

Abstract

We present TubeFormer-DeepLab, the first attempt to tackle multiple core video segmentation tasks in a unified manner. Different video segmentation tasks (e.g., video semantic/instance/panoptic segmentation) are usually considered as distinct problems. State-of-the-art models adopted in the separate communities have diverged, and radically different approaches dominate in each task. By contrast, we make a crucial observation that video segmentation tasks could be generally formulated as the problem of assigning different predicted labels to video tubes (where a tube is obtained by linking segmentation masks along the time axis) and the labels may encode different values depending on the target task. The observation motivates us to develop TubeFormer-DeepLab, a simple and effective video mask transformer model that is widely applicable to multiple video segmentation tasks. TubeFormer-DeepLab directly predicts video tubes with task-specific labels (either pure semantic categories, or both semantic categories and instance identities), which not only significantly simplifies video segmentation models, but also advances state-of-the-art results on multiple video segmentation benchmarks
BibTeX
@inproceedings{kim2022tubeformer,
  author = {Dahun Kim and Jun Xie and Huiyu Wang and Siyuan Qiao and Qihang Yu and Hong-Seok Kim and Hartwig Adam and In So Kweon and Liang-Chieh Chen},
  title = {{TubeFormer-DeepLab: Video Mask Transformer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure