Abstract: Unsupervised video object segmentation (VOS) aims to detect and segment the most salient object in videos. The primary techniques used in unsupervised VOS are 1) the collaboration of appearance and motion information and 2) temporal fusion between different frames. This paper proposes two novel prototype-based attention mechanisms, inter-modality attention (IMA) and inter-frame attention (IFA), to incorporate these techniques via dense propagation across different modalities and frames. IMA densely integrates context information from different modalities based on a mutual refinement. IFA injects global context of a video to the query frame, enabling a full utilization of useful properties from multiple frames. Experimental results on public benchmark datasets demonstrate that our proposed approach outperforms all existing methods by a substantial margin. The proposed two components are also thoroughly validated via ablative study.
Introduction
Unsupervised Video Object Segmentation (VOS) is the challenging task of identifying and segmenting the most salient object in a video sequence without any prior guidance, such as user-provided masks or text descriptions. Traditional approaches often struggle with two core issues: noisy multi-modal fusion and inefficient temporal modeling.
In multi-modal fusion, combining appearance (RGB) and motion (optical flow) cues naively (e.g., via addition or concatenation) can lead to performance degradation if one modality is noisy—for instance, when optical flow is blurred by fast motion. In temporal modeling, capturing long-range dependencies often requires expensive iterative processes or is limited to a few adjacent frames, failing to leverage the full global context of the video. This paper introduces the Dual Prototype Attention (DPA) framework to overcome these limitations.
Methodology
The proposed DPA framework utilizes a two-stream encoder-decoder architecture enhanced by two novel attention mechanisms: Inter-Modality Attention (IMA) and Inter-Frame Attention (IFA).
A key innovation is the use of a Prototype Framework. Instead of performing attention on high-dimensional pixel-level features, the model converts features into a compact set of "prototypes." These prototypes act as semantic anchors, filtering out local noise and significantly reducing computational complexity.
- Inter-Modality Attention (IMA): IMA facilitates mutual refinement between appearance and motion branches. By identifying semantic prototypes in both modalities, the branches "communicate" via cross-attention. This allows motion cues to refine appearance boundaries and vice versa, making the fusion robust to modality-specific noise.
- Inter-Frame Attention (IFA): To capture global temporal context efficiently, IFA samples a small number of frames across the entire video to build a global memory bank. It then uses prototype-based attention to propagate this global knowledge to the current query frame, helping the model handle occlusions and appearance changes over time.
Experimental Results
The DPA method was evaluated on standard benchmarks including DAVIS 2016, FBMS, and YouTube-Objects. On DAVIS 2016, it achieved a state-of-the-art mean $\mathcal{G}$ score of 87.6%, outperforming previous leaders. On FBMS, it reached a $\mathcal{J}$ mean of 83.4%. Despite the sophisticated attention modules, the model maintains high efficiency, running at approximately 19.5 to 24.2 FPS.
Ablation studies confirm the importance of both modules. As shown in the visualization above, the "Base" model's focus is often scattered or incomplete. Adding IMA helps localize the object better by aligning modalities, while IFA further refines the focus by incorporating temporal context from other frames. The combination of both (IMA and IFA) produces the most precise and robust saliency maps.
Conclusion
This work demonstrates that prototype-based attention is a powerful tool for unsupervised VOS. By abstracting features into semantic prototypes, IMA and IFA achieve robust multi-modal fusion and efficient global temporal aggregation. The framework sets a new performance standard while remaining computationally practical for real-world applications. Future work may involve improving the sampling strategy for IFA and reducing the reliance on external optical flow estimators.
Summary updated