Self-supervised Video Object Segmentation with Distillation Learning of Deformable Attention

Quang-Trung Truong, Duc Thanh Nguyen, Binh-Son Hua, Sai-Kit YeungarXiv 2024 · OpenReview

TL;DR

This paper introduces a novel self-supervised video object segmentation (VOS) method that leverages deformable attention and knowledge distillation. By allowing attention mechanisms to adapt to temporal changes and distilling knowledge from a larger teacher model, they achieve state-of-the-art performance with a lightweight architecture, making VOS more accessible for resource-constrained devices.

What Problem Does This Paper Address?

Video object segmentation (VOS) aims to precisely outline objects of interest across video frames. While attention mechanisms have shown promise in VOS by capturing object representations, they often struggle with the dynamic nature of videos. Temporal changes can cause attention maps to misalign with objects, leading to accumulated errors over time. Furthermore, many existing VOS methods rely on complex, computationally expensive architectures, hindering their deployment on low-power devices. This paper addresses these challenges by proposing a self-supervised approach that is both robust to temporal variations and computationally efficient.

Methods

The core of this work lies in two key innovations:

  • Deformable Attention Module: Inspired by deformable convolutions, this module enhances the standard self-attention mechanism. Instead of fixed locations for keys and values in the attention module, deformable attention learns flexible offsets. This allows the attention to adapt to spatial and temporal changes in the video, leading to more accurate object representation learning.
  • Knowledge Distillation with Attention Transfer: The authors propose a novel knowledge distillation (KD) framework for self-supervised VOS. They train a lightweight "student" network using pseudo-labels generated by a powerful "teacher" network (specifically, the DeAOT model). Crucially, their KD approach goes beyond traditional logit distillation by also transferring intermediate attention maps. This attention distillation, guided by a Centered Kernel Alignment (CKA) loss, helps the student network learn more robust and adaptive representations.

The overall architecture is designed to be lightweight, making it suitable for deployment on devices with limited computational resources.

Datasets

The proposed method was evaluated on standard VOS benchmark datasets:

  • DAVIS 2016 & 2017: These datasets focus on single object segmentation and provide ground truth for training and validation.
  • YouTube-VOS 2018 & 2019: These datasets feature multi-object segmentation and are larger in scale, with diverse categories.

For self-supervised training, the authors utilized synthetic video sequences generated from images from datasets like BIG, DUTS, ECSSD, FSS-1000, and HRSOD.

Key Insights and Ideas

  • Deformable Attention for Temporal Adaptability: The key insight is that by making attention locations learnable and dynamic (deformable), the model can better track objects through temporal changes and deformations, a significant limitation of fixed attention mechanisms.
  • Attention Distillation is Crucial for VOS: Simply distilling the final output (logits) from a teacher to a student isn't enough for VOS. Distilling intermediate attention maps provides a richer signal, enabling the student to learn more discriminative object representations that are robust to challenging scenarios.
  • Lightweight Architecture + KD = Efficiency: Combining a carefully designed lightweight student network with a sophisticated KD strategy allows for high performance without the computational burden of large models, opening doors for real-time VOS on edge devices.

Key Results

  • State-of-the-Art Performance: The proposed method achieves state-of-the-art performance on benchmark datasets like YouTube-VOS 2018 and DAVIS 2017, outperforming existing self-supervised VOS methods in terms of segmentation accuracy (J&F scores).
  • Significant Speedup and Memory Reduction: Compared to other high-performing methods, their approach offers a substantial speedup (e.g., ~50x faster than Mask-VOS) and significantly reduced memory footprint, making it practical for real-time applications and low-power devices.
  • Deformable Attention Superiority: Ablation studies show that the deformable attention module consistently outperforms vanilla attention across various metrics and datasets.
  • Effectiveness of Attention Distillation: The KD strategy that includes attention map distillation (using CKA loss) demonstrates superior performance compared to methods that only distill logits.

Limitations and Future Work

  • Intra-Target Scale Variations and Out-of-Plane Rotations: The method struggles with videos exhibiting significant intra-target scale variations (objects rapidly changing size) and out-of-plane rotations. This is attributed to the limitations of the teacher model's encoder/decoder in handling such drastic appearance changes, which then affects the knowledge transfer.
  • Future Directions: The authors suggest that their KD framework can be extended to other attention-based VOS models, indicating its general applicability. Further research could focus on improving robustness to the identified limitations.

Summary updated

Abstract

Video object segmentation is a fundamental research problem in computer vision. Recent techniques have often applied attention mechanism to object representation learning from video sequences. However, due to temporal changes in the video data, attention maps may not well align with the objects of interest across video frames, causing accumulated errors in long-term video processing. In addition, existing techniques have utilised complex architectures, requiring highly computational complexity and hence limiting the ability to integrate video object segmentation into low-powered devices. To address these issues, we propose a new method for self-supervised video object segmentation based on distillation learning of deformable attention. Specifically, we devise a lightweight architecture for video object segmentation that is effectively adapted to temporal changes. This is enabled by deformable attention mechanism, where the keys and values capturing the memory of a video sequence in the attention module have flexible locations updated across frames. The learnt object representations are thus adaptive to both the spatial and temporal dimensions. We train the proposed architecture in a self-supervised fashion through a new knowledge distillation paradigm where deformable attention maps are integrated into the distillation loss. We qualitatively and quantitatively evaluate our method and compare it with existing methods on benchmark datasets including DAVIS 2016/2017 and YouTube-VOS 2018/2019. Experimental results verify the superiority of our method via its achieved state-of-the-art performance and optimal memory usage.
BibTeX
@misc{truong2024self,
  author = {Quang-Trung Truong and Duc Thanh Nguyen and Binh-Son Hua and Sai-Kit Yeung},
  title = {{Self-supervised Video Object Segmentation with Distillation Learning of Deformable Attention}},
  year = {2024},
  eprint = {2401.13937},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.13937},
}
Figure