End-to-End Video Instance Segmentation with Transformers

Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, Huaxia Xia

Abstract

Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Recent methods typically develop sophisticated pipelines to tackle this task. Here, we propose a new video instance segmentation framework built upon Transformers, termed VisTR, which views the VIS task as a direct end-to-end parallel sequence decoding/prediction problem. Given a video clip consisting of multiple image frames as input, VisTR outputs the sequence of masks for each instance in the video in order directly. At the core is a new, effective instance sequence matching and segmentation strategy, which supervises and segments instances at the sequence level as a whole. VisTR frames the instance segmentation and tracking in the same perspective of similarity learning, thus considerably simplifying the overall pipeline. Without bells and whistles, VisTR achieves the highest speed among all existing VIS models and achieves the best result among methods using a single model on the YouTube-VIS dataset.

Introduction

Video Instance Segmentation (VIS) is a challenging computer vision task that requires a model to simultaneously classify, segment, and track every object of interest across all frames of a video. Traditionally, this has been approached through complex multi-stage pipelines. Top-down methods detect objects in individual frames and then use human-designed heuristics or separate re-identification branches to link them across time. Bottom-up methods cluster pixels based on learned embeddings, which often requires multiple iterative steps and is computationally expensive.

The authors of VisTR argue that these multi-stage pipelines are overly complex. They propose that because segmentation (pixel-level similarity) and tracking (instance-level similarity) are both essentially similarity learning problems, they should be solved together in a unified, end-to-end architecture. VisTR (Video Instance Segmentation TRansformer) is the first framework to treat VIS as a direct sequence-to-sequence prediction problem.

Methods

VisTR builds upon the success of the Detection Transformer (DETR) but extends it into the temporal domain. The architecture consists of four main components designed to handle the spatial and temporal dimensions of video data simultaneously.

The detailed architecture of VisTR, showing the flow from CNN backbone to transformer encoder-decoder and final sequence matching/segmentation modules.
  1. CNN Backbone: A standard ResNet extracts feature maps from a sequence of input frames. These features are concatenated to form a "clip-level" feature sequence.
  2. Transformer Encoder-Decoder: The encoder uses self-attention to model the relationships between all pixels across both space and time. The decoder takes "instance queries" to extract instance-level features. Crucially, the model predicts a sequence of $N$ outputs for each frame, where the order of predictions is kept consistent across frames to handle tracking naturally.
  3. Instance Sequence Matching: To train the model, the authors use a bipartite matching strategy (the Hungarian algorithm). Instead of matching individual boxes, they match entire sequences of predictions to ground-truth sequences, supervising the model at the video level.
  4. Instance Sequence Segmentation: To produce the final masks, the model accumulates features across frames and applies 3D Convolutions. This allows the model to reinforce mask consistency by looking at the object's appearance in neighboring frames.

A vital addition is the 3D Positional Encoding, which supplements the features with spatial (x, y) and temporal (t) information, ensuring the Transformer understands where and when a pixel occurs.

Key Insights

  • Tracking as Sequence Ordering: Tracking doesn't need a separate algorithm. If the Transformer decoder always outputs "Instance A" at index 1 and "Instance B" at index 2 for every frame, tracking is achieved simply by linking predictions with the same index.
  • Similarity is Universal: The self-attention mechanism in Transformers is uniquely suited for VIS because it naturally computes the similarity between any two points in a video, regardless of how far apart they are in time or space.
  • 3D Convolutions for Temporal Refinement: While the Transformer handles the logic of tracking, 3D convolutions ensure the segmentation masks are smooth and consistent across frames, effectively denoising the masks using temporal context.

Key Results

VisTR was evaluated on the YouTube-VIS dataset, a large-scale benchmark containing 2,883 high-resolution videos and 40 object categories. The model stands out primarily for its efficiency and simplicity:

  • Speed: It achieves 57.7 FPS on a ResNet-101 backbone, making it the fastest model among those using a single-model approach.
  • Accuracy: On the YouTube-VIS dataset, it achieved 40.1% mask mAP, outperforming the popular MaskTrack R-CNN by about 6 points and matching the performance of much more complex, multi-stage systems like MaskProp.
  • Qualitative Success: The model proved robust in challenging scenarios, such as when objects overlap, change poses drastically, or are momentarily occluded.
Qualitative results of VisTR on various challenging video sequences, demonstrating consistent tracking and segmentation.

Limitations and Future Work

Despite its success, VisTR faces memory constraints due to the quadratic computational complexity of Transformers relative to sequence length. This limits the temporal window the model can process at once (currently 36 frames). Processing very long, high-resolution videos remains a challenge.

Furthermore, the authors intentionally avoided "bells and whistles" like complex data augmentation to prove the core concept. Future work may involve adding these standard computer vision refinements to further boost accuracy and extending the framework to other video-level tasks like action recognition or video object segmentation (VOS).

Summary updated

Abstract

Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Recent methods typically develop sophisticated pipelines to tackle this task. Here, we propose a new video instance segmentation framework built upon Transformers, termed VisTR, which views the VIS task as a direct end-to-end parallel sequence decoding/prediction problem. Given a video clip consisting of multiple image frames as input, VisTR outputs the sequence of masks for each instance in the video in order directly. At the core is a new, effective instance sequence matching and segmentation strategy, which supervises and segments instances at the sequence level as a whole. VisTR frames the instance segmentation and tracking in the same perspective of similarity learning, thus considerably simplifying the overall pipeline and is significantly different from existing approaches. Without bells and whistles, VisTR achieves the highest speed among all existing VIS models, and achieves the best result among methods using single model on the YouTube-VIS dataset. For the first time, we demonstrate a much simpler and faster video instance segmentation framework built upon Transformers, achieving competitive accuracy. We hope that VisTR can motivate future research for more video understanding tasks.
BibTeX
@inproceedings{wang2021end,
  author = {Yuqing Wang and Zhaoliang Xu and Xinlong Wang and Chunhua Shen and Baoshan Cheng and Hao Shen and Huaxia Xia},
  title = {{End-to-End Video Instance Segmentation with Transformers}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=dOp4ZJxUTD-},
}
Figure