Unsupervised Video Object Segmentation via Prototype Memory Network

Minhyeok Lee, Suhwan Cho, Seunghoon Lee, Chaewon Park, Sangyoun Lee

Abstract

Unsupervised video object segmentation aims to segment a target object in the video without a ground truth mask in the initial frame. This challenging task requires extracting features for the most salient common objects within a video sequence. This difficulty can be solved by using motion information such as optical flow, but using only the information between adjacent frames results in poor connectivity between distant frames and poor performance. To solve this problem, we propose a novel prototype memory network architecture. The proposed model effectively extracts the RGB and motion information by extracting superpixel-based component prototypes from the input RGB images and optical flow maps. In addition, the model scores the usefulness of the component prototypes in each frame based on a self-learning algorithm and adaptively stores the most useful prototypes in memory and discards obsolete prototypes. We use the prototypes in the memory bank to predict the next query frame’s mask, which enhances the association between distant frames to help with accurate mask prediction. Our method is evaluated on three datasets, achieving state-of-the-art performance. We prove the effectiveness of the proposed model with various ablation studies.

Introduction

Video Object Segmentation (VOS) involves identifying and outlining salient objects at a pixel level. In the unsupervised setting (UVOS), the challenge is magnified as the model must autonomously identify the primary object without initial guidance. Traditional approaches rely heavily on optical flow to capture motion, but these methods often suffer from "short-sightedness," failing when objects stop moving or become occluded over long sequences. Furthermore, effectively fusing appearance (RGB) and motion (Flow) data remains a significant hurdle.

Proposed Methodology

The Prototype Memory Network (PMN) addresses these challenges by representing objects as a collection of "component prototypes."

Architectural Overview

The PMN architecture processes both RGB and motion (optical flow) information through parallel streams. It utilizes dual encoders that feed into Prototype Generation Modules (PGM). These modules use the SLIC algorithm to group pixels into superpixels, ensuring that generated prototypes represent coherent object parts rather than arbitrary grid regions.

Detailed architecture of the Prototype Memory Network (PMN), showing the integration of RGB and Flow sequences through Prototype Generation Modules (PGM) and Prototype Scoring Modules (PSM).

Prototype Scoring and Memory Management

A key innovation is the Prototype Scoring Module (PSM), a self-supervised Transformer-based component that evaluates the "usefulness" of each prototype. By identifying correlations between prototypes that exhibit consistent foreground characteristics, the PSM filters out noise. High-scoring prototypes are stored in a persistent memory bank, allowing the model to bridge temporal gaps and maintain object identity even during periods of occlusion or stationary behavior.

The temporal workflow of PMN, illustrating how prototypes are scored, stored in a memory bank, and used for mask prediction across successive frames.

Mask Prediction

The Correlation Map Generating Module (CMGM) produces the final segmentation by comparing current frame features against the memory bank using pixel-wise cosine similarity. This global comparison ensures that the prediction is informed by the object's entire history within the video.

Experimental Results

The PMN was evaluated on three major benchmarks: DAVIS-16, FBMS, and YouTube-Objects. The model achieved state-of-the-art (SOTA) performance, notably reaching a $\mathcal{J}\&\mathcal{F}$ mean score of 85.9 on DAVIS-16.

Dataset Metric PMN Score
DAVIS-16 $\mathcal{J}\&\mathcal{F}$ Mean 85.9
FBMS $\mathcal{J}$ Mean 77.8
YouTube-Objects $\mathcal{J}$ Mean 71.8

Qualitative analysis confirms the model's robustness against background distractors and extreme scale changes, such as fast-moving objects or complex occlusions.

Conclusion

The Prototype Memory Network introduces a robust framework for UVOS by leveraging superpixel-based prototypes and a self-supervised memory mechanism. By moving beyond frame-to-frame motion analysis, PMN achieves superior long-term object consistency. Future work may involve reducing the computational overhead of the memory bank and extending the architecture to handle multi-object unsupervised segmentation.

Summary updated

Abstract

Unsupervised video object segmentation aims to segment a target object in the video without a ground truth mask in the initial frame. This challenging task requires extracting features for the most salient common objects within a video sequence. This difficulty can be solved by using motion information such as optical flow, but using only the information between adjacent frames results in poor connectivity between distant frames and poor performance. To solve this problem, we propose a novel prototype memory network architecture. The proposed model effectively extracts the RGB and motion information by extracting superpixel-based component prototypes from the input RGB images and optical flow maps. In addition, the model scores the usefulness of the component prototypes in each frame based on a self-learning algorithm and adaptively stores the most useful prototypes in memory and discards obsolete proto-types. We use the prototypes in the memory bank to predict the next query frame’s mask, which enhances the association between distant frames to help with accurate mask prediction. Our method is evaluated on three datasets, achieving state-of-the-art performance. We prove the effectiveness of the proposed model with various ablation studies.
BibTeX
@inproceedings{lee2023unsupervised,
  author = {Minhyeok Lee and Suhwan Cho and Seunghoon Lee and Chaewon Park and Sangyoun Lee},
  title = {{Unsupervised Video Object Segmentation via Prototype Memory Network}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=zORNPmZ5eIt},
}
Figure