Improving Video Instance Segmentation via Temporal Pyramid Routing

Xiangtai Li, Hao He, Henghui Ding, Kuiyuan Yang, Guangliang Cheng, Jianping Shi, Yunhai Tong

TL;DR

This paper introduces Temporal Pyramid Routing (TPR), a novel module for Video Instance Segmentation (VIS) that effectively addresses the challenges of temporal and scale variations in videos. TPR dynamically aligns and aggregates features from adjacent frames across different scales, significantly improving instance segmentation and tracking accuracy while remaining computationally efficient and plug-and-play.

What Problem Does This Paper Address?

Video Instance Segmentation (VIS) is a complex task that requires not only detecting and segmenting individual objects within a single frame but also tracking them consistently across a video sequence. Existing methods often struggle with two key challenges: temporal inconsistencies (how to effectively use information from previous frames) and scale variations (how objects can change size dramatically between frames). Many approaches either rely on single-frame features or simple multi-frame feature aggregation, neglecting the crucial interplay between temporal context and multi-scale representations. This leads to difficulties in accurately segmenting and tracking objects, especially when they undergo significant appearance changes or overlap with other instances.

Methods

The core contribution of this paper is the Temporal Pyramid Routing (TPR) strategy. TPR is designed to be a lightweight, plug-and-play module that can be integrated into existing instance segmentation frameworks. It operates on feature pyramids from two adjacent frames (a query frame and a reference frame) and refines the query frame's features by incorporating relevant information from the reference frame. TPR consists of two novel components:

  • Dynamic Aligned Cell Routing (DACR): This component focuses on aligning and gating features across the temporal dimension. It uses a "sample and filter" strategy. First, it employs dynamic sampling (similar to deformable convolutions) to find correspondences between pixels in the query and reference frames. Then, it uses a novel "double gate" mechanism:
  • Inner Gates: These gates filter out background noise and identify fine-grained, relevant semantic information from the reference frame.
  • Outer Gates: These gates highlight important regions in the query frame and selectively absorb features from the reference frame, further refining the alignment and preventing the propagation of irrelevant content.
  • Cross Pyramid Routing (CPR): This component transfers the temporally aggregated features across different scales of the feature pyramid. It uses a routing space that is designed to be "bottom-up" (from lower-level to higher-level features), which the authors argue is more effective for bridging semantic gaps and handling misalignments in lower resolutions. This contrasts with traditional top-down FPN structures.

The TPR module is designed to be efficient, dynamically adjusting computation based on input features, and can be seamlessly integrated into existing VIS pipelines.

Datasets

The proposed method was evaluated on the following datasets:

  • YouTube-VIS-2019: A standard benchmark for Video Instance Segmentation.
  • YouTube-VIS-2021: An improved and augmented version of the YouTube-VIS-2019 dataset.
  • Cityscapes-VPS: Used for evaluating Video Panoptic Segmentation (VPS).

Key Insights and Ideas

  • Temporal and Scale Information are Crucial and Intertwined: The paper strongly emphasizes that effectively handling both temporal dynamics and scale variations is essential for robust VIS, and that these two aspects are not independent.
  • Dynamic Routing for Efficiency and Adaptability: Instead of static feature aggregation, the authors leverage dynamic routing, inspired by image-based dynamic networks. This allows the network to adapt its computation and feature selection based on the input, leading to better performance and efficiency.
  • The "Double Gate" is Key for Temporal Alignment: The DACR's inner and outer gates are crucial for selectively absorbing relevant temporal information while discarding noise and irrelevant background, a significant improvement over simpler warping or aggregation methods.
  • Bottom-Up Cross-Scale Routing is Advantageous: The CPR's bottom-up routing strategy is shown to be more effective than top-down approaches for propagating aligned features across scales, particularly for bridging semantic gaps.
  • Plug-and-Play Modularity: The TPR's ability to be easily integrated into various existing VIS frameworks without requiring fundamental architectural changes is a major practical advantage.

Key Results

  • Significant Accuracy Improvements: The proposed TPR module consistently improves the performance of various baseline VIS methods. For example, it achieves a 2.1%-2.5% mAP gain over static baselines on YouTube-VIS-2019.
  • State-of-the-Art Performance: With a Swin-base backbone, the method achieves 45.9 mAP on YouTube-VIS-2019 in the online inference setting, outperforming MaskProp while being significantly faster.
  • Efficiency: TPR is computationally efficient, offering substantial accuracy gains with only a small increase in GFlops (around 6.4%-7.7% average increase). It is also reported to be about 10 times faster than MaskProp.
  • Generalization: The TPR module demonstrates strong generalization capabilities, improving performance when applied to different instance segmentation approaches like BlendMask, SipMask, MaskTrack-RCNN, and YOLACT.
  • VPS Performance: The method also shows promise on Video Panoptic Segmentation (VPS), achieving gains on the Cityscapes-VPS dataset.

Limitations and Future Work

  • Computational Cost: While TPR is efficient, it still introduces some computational overhead compared to single-frame methods. The paper discusses managing this through budget loss, but it remains a consideration for extremely resource-constrained environments.
  • Reference Frame Selection: The paper mentions using one previous reference frame for inference. Future work could explore more sophisticated strategies for selecting or aggregating information from multiple reference frames.
  • Dynamic Network Design: The authors suggest that their work provides insights into dynamic network design for video tasks, implying that further exploration in this area could lead to even more advanced solutions.
  • Application to Other Video Tasks: The success on VPS suggests potential for TPR in other video understanding tasks, which could be a direction for future research.

Summary updated

Abstract

Video Instance Segmentation (VIS) is a new and inherently multi-task problem, which aims to detect, segment, and track each instance in a video sequence. Existing approaches are mainly based on single-frame features or single-scale features of multiple frames, where either temporal information or multi-scale information is ignored. To incorporate both temporal and scale information, we propose a Temporal Pyramid Routing (TPR) strategy to conditionally align and conduct pixel-level aggregation from a feature pyramid pair of two adjacent frames. Specifically, TPR contains two novel components, including Dynamic Aligned Cell Routing (DACR) and Cross Pyramid Routing (CPR), where DACR is designed for aligning and gating pyramid features across temporal dimension, while CPR transfers temporally aggregated features across scale dimension. Moreover, our approach is a light-weight and plug-and-play module and can be easily applied to existing instance segmentation methods. Extensive experiments on three datasets including YouTube-VIS (2019, 2021) and Cityscapes-VPS demonstrate the effectiveness and efficiency of the proposed approach on several state-of-the-art video instance and panoptic segmentation methods. Codes will be publicly available at \url{https://github.com/lxtGH/TemporalPyramidRouting}.
BibTeX
@article{li2023improving,
  author = {Xiangtai Li and Hao He and Henghui Ding and Kuiyuan Yang and Guangliang Cheng and Jianping Shi and Yunhai Tong},
  title = {{Improving Video Instance Segmentation via Temporal Pyramid Routing}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=9SPIxoMPsA8},
}
Figure