Dual Prototype Attention for Unsupervised Video Object Segmentation

Suhwan Cho, Minhyeok Lee, Seunghoon Lee, Dogyoon Lee, Heeseung Choi, Ig-Jae Kim, Sangyoun Lee

Introduction

Unsupervised Video Object Segmentation (VOS) aims to identify and segment the most salient object in a video sequence without prior knowledge or manual initialization. Achieving high performance in this task requires effectively leveraging multi-modal data—specifically appearance (RGB) and motion (optical flow)—while maintaining temporal coherence across frames.

The Problem: Inefficient Fusion and Limited Context

Existing VOS methods often struggle with two primary challenges. First, the fusion of appearance and motion features is frequently handled through "naive" operations like simple addition or concatenation. This makes models susceptible to noise; for instance, a blurry optical flow map can degrade the overall segmentation quality. Second, many models lack a global temporal perspective, either focusing only on adjacent frames or becoming computationally prohibitive when attempting to process the entire video sequence at once.

Proposed Method: Dual Prototype Attention (DPA)

To address these limitations, the authors introduce the Dual Prototype Attention (DPA) framework. The core innovation lies in the use of "prototypes"—compact, representative semantic clusters—rather than raw pixel-level features for attention mechanisms.

Overview of the DPA architecture, showing the integration of Inter-Modality Attention (IMA) and Inter-Frame Attention (IFA) modules within a two-stream encoder-decoder backbone.

The DPA architecture consists of two primary modules:

Inter-Modality Attention (IMA)

IMA facilitates mutual refinement between the appearance and motion branches. Instead of direct fusion, IMA extracts prototypes from both modalities. It then employs cross-attention, allowing the motion data to guide the appearance branch toward moving objects while the appearance data helps the motion branch filter out background noise. This "mutual supervision" ensures that the model remains robust even when one modality is noisy.

Inter-Frame Attention (IFA)

To capture global context efficiently, IFA samples a small set of representative frames from across the entire video. These frames are stored as prototypes in a global memory bank. When processing a query frame, the model "reads" from this memory to maintain consistency, which is particularly useful for handling occlusions where the object might temporarily disappear.

Key Insights: The Power of Prototypes

The transition from pixel-level attention to prototype-level attention provides a natural denoising effect. By condensing complex image data into a few dozen semantic clusters, the attention mechanism becomes both more robust to local noise and significantly more computationally efficient.

Visual impact of IMA and IFA on attention maps. The combination of both modules (bottom right) leads to a much cleaner and more accurate focus on the salient object compared to the base model.

Furthermore, the authors demonstrate that "global context is cheap." Sampling just 3 to 5 representative frames provides sufficient temporal information to boost performance significantly without the latency associated with processing every frame.

Experimental Results

The DPA model was evaluated on several major benchmarks, setting new state-of-the-art records:

  • DAVIS 2016: Achieved a $\mathcal{G}_M$ score of 87.6%, outperforming previous leading methods like HFAN and TMO.
  • FBMS: Demonstrated superior robustness with a $\mathcal{J}_M$ of 83.4%, highlighting its ability to handle complex motion.
  • Efficiency: The model maintains high inference speeds, running at 19.5 to 24.2 FPS, making it suitable for near-real-time applications.

Conclusion and Future Work

Dual Prototype Attention represents a significant step forward in unsupervised VOS by providing a robust and efficient way to handle multi-modal fusion and global temporal context. Future research may explore more sophisticated "intelligent" frame sampling strategies for IFA or adapt the prototype approach for more complex multi-object tracking and segmentation (MOTS) tasks.

Summary updated

Abstract

Unsupervised video object segmentation (VOS) aims to detect and segment the most salient object in videos. The primary techniques used in unsupervised VOS are 1) the collaboration of appearance and motion information and 2) temporal fusion between different frames. This paper proposes two novel prototype-based attention mechanisms, inter-modality attention (IMA) and inter-frame attention (IFA), to incorporate these techniques via dense propagation across different modalities and frames. IMA densely integrates context information from different modalities based on a mutual refinement. IFA injects global context of a video to the query frame, enabling a full utilization of useful properties from multiple frames. Experimental results on public benchmark datasets demonstrate that our proposed approach outperforms all existing methods by a substantial margin. The proposed two components are also thoroughly validated via ablative study.
BibTeX
@inproceedings{cho2024dual,
  author = {Suhwan Cho and Minhyeok Lee and Seunghoon Lee and Dogyoon Lee and Heeseung Choi and Ig-Jae Kim and Sangyoun Lee},
  title = {{Dual Prototype Attention for Unsupervised Video Object Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01820},
}
Figure