Abstract
Unsupervised video object segmentation aims to segment a target object in the video without a ground truth mask in the initial frame. This challenging task requires extracting features for the most salient common objects within a video sequence. This difficulty can be solved by using motion information such as optical flow, but using only the information between adjacent frames results in poor connectivity between distant frames and poor performance. To solve this problem, we propose a novel prototype memory network architecture. The proposed model effectively extracts the RGB and motion information by extracting superpixel-based component prototypes from the input RGB images and optical flow maps. In addition, the model scores the usefulness of the component prototypes in each frame based on a self-learning algorithm and adaptively stores the most useful prototypes in memory and discards obsolete prototypes. We use the prototypes in the memory bank to predict the next query frame’s mask, which enhances the association between distant frames to help with accurate mask prediction. Our method is evaluated on three datasets, achieving state-of-the-art performance. We prove the effectiveness of the proposed model with various ablation studies.
Introduction
Video Object Segmentation (VOS) involves identifying and outlining salient objects at a pixel level. In the unsupervised setting (UVOS), the challenge is magnified as the model must autonomously identify the primary object without initial guidance. Traditional approaches rely heavily on optical flow to capture motion, but these methods often suffer from "short-sightedness," failing when objects stop moving or become occluded over long sequences. Furthermore, effectively fusing appearance (RGB) and motion (Flow) data remains a significant hurdle.
Proposed Methodology
The Prototype Memory Network (PMN) addresses these challenges by representing objects as a collection of "component prototypes."
Architectural Overview
The PMN architecture processes both RGB and motion (optical flow) information through parallel streams. It utilizes dual encoders that feed into Prototype Generation Modules (PGM). These modules use the SLIC algorithm to group pixels into superpixels, ensuring that generated prototypes represent coherent object parts rather than arbitrary grid regions.
Prototype Scoring and Memory Management
A key innovation is the Prototype Scoring Module (PSM), a self-supervised Transformer-based component that evaluates the "usefulness" of each prototype. By identifying correlations between prototypes that exhibit consistent foreground characteristics, the PSM filters out noise. High-scoring prototypes are stored in a persistent memory bank, allowing the model to bridge temporal gaps and maintain object identity even during periods of occlusion or stationary behavior.
Mask Prediction
The Correlation Map Generating Module (CMGM) produces the final segmentation by comparing current frame features against the memory bank using pixel-wise cosine similarity. This global comparison ensures that the prediction is informed by the object's entire history within the video.
Experimental Results
The PMN was evaluated on three major benchmarks: DAVIS-16, FBMS, and YouTube-Objects. The model achieved state-of-the-art (SOTA) performance, notably reaching a $\mathcal{J}\&\mathcal{F}$ mean score of 85.9 on DAVIS-16.
| Dataset | Metric | PMN Score |
|---|---|---|
| DAVIS-16 | $\mathcal{J}\&\mathcal{F}$ Mean | 85.9 |
| FBMS | $\mathcal{J}$ Mean | 77.8 |
| YouTube-Objects | $\mathcal{J}$ Mean | 71.8 |
Qualitative analysis confirms the model's robustness against background distractors and extreme scale changes, such as fast-moving objects or complex occlusions.
Conclusion
The Prototype Memory Network introduces a robust framework for UVOS by leveraging superpixel-based prototypes and a self-supervised memory mechanism. By moving beyond frame-to-frame motion analysis, PMN achieves superior long-term object consistency. Future work may involve reducing the computational overhead of the memory bank and extending the architecture to handle multi-object unsupervised segmentation.
Summary updated