Abstract
In this paper we introduce a Transformer-based approach to video object segmentation (VOS). To address compounding error and scalability issues of prior work, we propose a scalable, end-to-end method for VOS called Sparse Spatiotemporal Transformers (SST). SST extracts per-pixel representations for each object in a video using sparse attention over spatiotemporal features. Our attention-based formulation for VOS allows a model to learn to attend over a history of multiple frames and provides suitable inductive bias for performing correspondence-like computations necessary for solving motion segmentation. We demonstrate the effectiveness of attention-based over recurrent networks in the spatiotemporal domain. Our method achieves competitive results on YouTube-VOS and DAVIS 2017 with improved scalability and robustness to occlusions compared with the state of the art.
Introduction
Video Object Segmentation (VOS) is the task of tracking and segmenting specific objects across a video sequence, given only the mask of the object in the first frame. Historically, VOS methods have relied on either slow online finetuning or recurrent/matching networks. However, Recurrent Neural Networks (RNNs) suffer from "compounding error," where small mistakes in early frames are magnified over time. While standard "dense" Transformers are excellent at capturing long-range dependencies, their computational cost scales quadratically with the number of pixels and frames ($O((THW)^2)$), making them impractical for video.
SSTVOS addresses these issues by introducing a scalable Transformer-based architecture that replaces dense attention with sparse spatiotemporal attention, providing global reasoning without the catastrophic computational cost.
Methods: Sparse Spatiotemporal Transformers
The core innovation of SST is the factorization of the attention mechanism to avoid the $O(N^2)$ bottleneck. The authors propose two primary sparse attention variants:
- Grid Attention: Instead of every pixel attending to every other pixel in the 3D video volume, a pixel only attends to others along its own X, Y, and T (time) axes. Stacking these layers allows information to propagate throughout the entire volume.
- Strided Attention: This decomposes attention into "local" components (attending to a small nearby cube of pixels) and "strided" components (attending to pixels at fixed intervals across the whole video).
The architecture utilizes a ResNet-101 backbone to extract features, which are then processed by a Transformer encoder to produce an Object Affinity Tensor. This tensor represents how pixels in the current frame correspond to pixels in the history of past frames. A convolutional decoder then uses these affinities to project the final segmentation mask.
Experimental Results
SSTVOS was evaluated on the YouTube-VOS (2018 & 2019) and DAVIS 2017 benchmarks. The model demonstrated significant improvements in both efficiency and accuracy.
Efficiency and Scalability
By factorizing 3D attention, SST (Grid) reduces the computational overhead from 66.6% (naive attention) to just 0.6% relative to the backbone. SST (Grid) uses only 1.45 GFLOPs for its attention layers, compared to 170.1 GFLOPs for naive dense attention.
Benchmarking Performance
On YouTube-VOS 2019, SST achieved an overall score ($\mathcal{G}$) of 81.8%, outperforming the previous state-of-the-art (CFBI). The model performed exceptionally well on "unseen" object categories, indicating strong generalization capabilities.
Conclusion
SSTVOS introduces a robust, scalable approach to VOS by leveraging sparse spatiotemporal Transformers. By treating attention as a correspondence-like computation, the model effectively handles long-range dependencies and occlusions while maintaining high computational efficiency. Future work may involve incorporating long-form Transformer techniques to handle even longer video sequences beyond the current sliding window history.
Summary updated