Unsupervised Video Object Segmentation via Prototype Memory Network

Minhyeok Lee, Suhwan Cho, Seunghoon Lee, Chaewon Park, Sangyoun Lee

Abstract

Unsupervised video object segmentation aims to segment a target object in the video without a ground truth mask in the initial frame. This challenging task requires extracting features for the most salient common objects within a video sequence. This difficulty can be solved by using motion information such as optical flow, but using only the information between adjacent frames results in poor connectivity between distant frames and poor performance. To solve this problem, we propose a novel prototype memory network architecture. The proposed model effectively extracts the RGB and motion information by extracting superpixel-based component prototypes from the input RGB images and optical flow maps. In addition, the model scores the usefulness of the component prototypes in each frame based on a self-learning algorithm and adaptively stores the most useful prototypes in memory and discards obsolete prototypes. We use the prototypes in the memory bank to predict the next query frames mask, which enhances the association between distant frames to help with accurate mask prediction. Our method is evaluated on three datasets, achieving state-of-the-art performance. We prove the effectiveness of the proposed model with various ablation studies.

Introduction

Video Object Segmentation (VOS) is a fundamental task in computer vision that requires pixel-level tracking of objects across video sequences. In the unsupervised setting, the challenge is amplified: the model must autonomously identify and segment the most "salient" or eye-catching common object across the sequence without being provided a ground-truth mask for the initial frame.

Existing approaches often leverage optical flow to capture motion cues. However, these methods typically suffer from two major limitations: short-range dependency, where looking only at adjacent frames leads to tracking failure during occlusions, and the domain gap, which makes the effective fusion of static RGB appearance with dynamic motion information difficult. The Prototype Memory Network (PMN) addresses these issues by representing objects as a collection of "component prototypes" and utilizing a memory bank to maintain long-term consistency.

Methodology

The Prototype Memory Network (PMN) architecture is designed to bridge the gap between appearance and motion while ensuring long-range temporal connectivity. The system consists of four primary modules:

Prototype Generating Module (PGM)

Rather than extracting a single global feature for an object, the PGM divides the input image into superpixels—clusters of pixels with similar visual characteristics. By extracting features from these local regions, the model creates "component prototypes." This granular approach allows the network to remember specific parts of an object (e.g., a wheel vs. a car door), providing robustness against partial occlusions.

Prototype Scoring Module (PSM)

To distinguish between salient object parts and background noise, the PSM employs a Transformer block to evaluate the "usefulness" of each prototype. Since explicit labels for prototype utility are unavailable, the module is trained via a self-supervised correlation mechanism, determining which features consistently represent the target across time.

Overview of the PMN architecture showing the dual-stream processing of RGB and Flow data through PGM, PSM, and the memory bank.

Memory Bank and Correlation Map Generation

The highest-scoring prototypes are stored in a dynamic memory bank. As the video progresses, the bank is updated to discard obsolete features and retain those that consistently represent the salient object. To predict the final segmentation mask, the Correlation Map Generating Module (CMGM) computes the pixel-wise cosine similarity between the current frame's features and the prototypes stored in the memory bank. This similarity map is then processed by a decoder to produce the predictive mask sequence.

Experimental Results

The PMN was evaluated on three standard unsupervised VOS benchmarks, achieving state-of-the-art (SOTA) performance across the board:

  • DAVIS-16: Reached a J&F score of 85.9, outperforming previous leaders such as IMP (85.6) and RTNet (85.2).
  • FBMS (Freiburg-Berkeley Motion Segmentation): Achieved a J-Mean of 77.8, a significant improvement over AMC-Net (76.5).
  • YouTube-Objects: Achieved a J-Mean of 71.8, setting a new high bar for this large-scale dataset.
Qualitative comparison of PMN against other state-of-the-art methods on the Breakdance, Bmx-Trees, and Motocross-Jump sequences.

Qualitative analysis demonstrates that PMN is particularly robust in sequences involving extreme scale changes and rapid morphological variations, such as the "breakdance" and "motocross" examples shown above. By maintaining a memory of object components, the model successfully tracks the target even when its appearance changes drastically.

Discussion and Conclusion

A key insight of this work is the transition from global object prototypes to superpixel-based component prototypes. This allows the model to handle partial occlusions effectively; if one part of an object is hidden, the memory bank still contains prototypes for the visible parts. Furthermore, by treating both RGB and Optical Flow features as prototypes within the same memory architecture, the model bypasses traditional multi-modal fusion difficulties.

Despite its success, the model's performance remains partially dependent on the quality of the underlying optical flow estimation (e.g., RAFT). Future work may involve reducing the computational overhead associated with superpixel generation and extending the framework to handle multi-object unsupervised segmentation, which remains a complex frontier in the field.

Summary updated

Abstract

Unsupervised video object segmentation aims to segment a target object in the video without a ground truth mask in the initial frame. This challenging task requires extracting features for the most salient common objects within a video sequence. This difficulty can be solved by using motion information such as optical flow, but using only the information between adjacent frames results in poor connectivity between distant frames and poor performance. To solve this problem, we propose a novel prototype memory network architecture. The proposed model effectively extracts the RGB and motion information by extracting superpixel-based component prototypes from the input RGB images and optical flow maps. In addition, the model scores the usefulness of the component prototypes in each frame based on a self-learning algorithm and adaptively stores the most useful prototypes in memory and discards obsolete prototypes. We use the prototypes in the memory bank to predict the next query frames mask, which enhances the association between distant frames to help with accurate mask prediction. Our method is evaluated on three datasets, achieving state-of-the-art performance. We prove the effectiveness of the proposed model with various ablation studies.
BibTeX
@inproceedings{lee2023unsupervised,
  author = {Minhyeok Lee and Suhwan Cho and Seunghoon Lee and Chaewon Park and Sangyoun Lee},
  title = {{Unsupervised Video Object Segmentation via Prototype Memory Network}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=zORNPmZ5eIt},
}
Figure