Temporal Context Enhanced Referring Video Object Segmentation

Xiao Hu, Basavaraj Hampiholi, Heiko Neumann, Jochen LangWACV 2024

TL;DR

This paper introduces TCE-RVOS, a novel approach to referring video object segmentation that significantly improves temporal context understanding. By introducing a "frame token fusion encoder" and an "instance query transformer decoder," TCE-RVOS effectively captures inter-frame relationships, leading to state-of-the-art performance, especially in challenging scenarios like occlusions and motion blur.

What Problem Does This Paper Address?

Referring Video Object Segmentation (RVOS) is a challenging cross-modal task that requires segmenting a specific object in a video based on a natural language description. While previous methods have leveraged transformers to fuse visual and textual information, they often overlook the crucial temporal dynamics within videos. This lack of temporal understanding leads to difficulties in handling situations like motion blur, occlusions, and data association across frames, where an object's appearance or visibility changes over time. Existing methods tend to process frames somewhat independently, hindering their ability to build a coherent understanding of the object's trajectory and context.

Methods

TCE-RVOS tackles the temporal context limitation through two main novel components:

  • Frame Token Fusion (FTF) Encoder: This module is designed to aggregate features across frames. It uses "memory tokens" that first distill information from each frame individually and then enrich the overall encoding by incorporating information from other frames. This allows for a more holistic understanding of temporal dynamics. The FTF encoder utilizes deformable self-attention and cross-attention mechanisms to efficiently process multi-scale features and reduce memory usage.
  • Instance Query Transformer (IQT) Decoder: This component focuses on aggregating queries related to the same object across different frames. It establishes a two-step feature aggregation structure: first, instance queries aggregate spatial features from each frame independently, and then temporal features from all frames belonging to the same instance are fused. This is achieved by rearranging and grouping instance queries based on their corresponding instance, followed by self-attention to capture temporal context. This strategy helps overcome issues caused by insufficient visual information in individual frames, particularly during occlusions or motion blur.

The overall framework also includes a standard visual backbone (like ResNet or Video Swin Transformer), a linguistic backbone (RoBERTa), and a vision-text early fusion block, followed by the proposed FTF encoder and IQT decoder.

Datasets

The paper evaluates TCE-RVOS on the following datasets:

  • Ref-Youtube-VOS [35]: A large-scale dataset with high-resolution videos and numerous language expressions. The authors note that the test set is not publicly available, so they evaluated on the validation set. They also introduce a new classification of this validation set into subcategories (occlusion, motion, crowded, etc.) to better analyze performance on challenging cases.
  • A2D-Sentences [12]: This dataset extends the original A2D video object segmentation dataset with text expressions.
  • Ref-DAVIS17 [20]: A smaller dataset built from DAVIS17 by adding text expressions.

Key Insights and Ideas

  • Temporal context is key for robust RVOS: The paper strongly emphasizes that neglecting temporal information is a major bottleneck for current RVOS methods, especially in handling dynamic scenarios.
  • Memory tokens for inter-frame distillation: The FTF encoder's use of memory tokens to distill and aggregate information across frames is a novel way to capture temporal dependencies without excessive computational cost.
  • Instance-centric temporal aggregation: The IQT decoder's approach of grouping and processing instance queries across frames directly addresses the challenge of maintaining object identity and context over time.
  • Improved handling of challenging scenarios: The proposed method shows a marked improvement in scenarios involving occlusion and motion blur, demonstrating the practical benefits of enhanced temporal understanding.
  • Dataset analysis for targeted improvement: The authors' classification of the Ref-Youtube-VOS validation set into specific challenging scenarios provides valuable insights into where and why their method excels.

Key Results

TCE-RVOS achieves state-of-the-art performance on several benchmarks:

  • Ref-Youtube-VOS:
    • With a ResNet-50 backbone, TCE-RVOS improves the J&F score by 4.0 points over the previous state-of-the-art (ReferFormer [39]).
  • With a VSwin-Tiny backbone, it achieves a 1.9-point improvement.
  • It also outperforms models using larger backbones like VSwin-Small.
  • A2D-Sentences: TCE-RVOS achieves a +2.0 mAP improvement using the VSwin-Tiny backbone.
  • Ref-DAVIS17: The model shows good generalization, achieving similar accuracy to ReferFormer without fine-tuning on this dataset.
  • Ablation Studies: Both the FTF encoder and IQT decoder individually contribute to performance gains. Notably, the FTF encoder shows a greater impact on contour accuracy (F) than region similarity (J). The performance consistently improves with an increased temporal window size (up to 5 frames).

Limitations and Future Work

  • Real-time performance: The paper acknowledges that the model, like its baseline ReferFormer, is not suitable for real-time applications, with inference taking around 5 seconds for a 10-frame video.
  • Vision-text aggregation: While temporal context is significantly improved, there is still room for enhancement in the vision-text aggregation, particularly in handling ambiguous text descriptions.
  • Future directions: The authors suggest exploring improvements in real-time processing and further enhancing the model's ability to handle ambiguous language.

Summary updated

BibTeX
@inproceedings{hu2024temporal,
  author = {Xiao Hu and Basavaraj Hampiholi and Heiko Neumann and Jochen Lang},
  title = {{Temporal Context Enhanced Referring Video Object Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure