Video Object Segmentation with Dynamic Memory Networks and Adaptive Object Alignment

Shuxian Liang, Xu Shen, Jianqiang Huang, Xian-Sheng Hua

Abstract

Video Object Segmentation (VOS) involves tracking and segmenting specific objects across video sequences. Current approaches are often divided into pixel-matching and object-matching methods. While object-matching is efficient, it historically struggled with evolving object appearances and spatial misalignments. This paper introduces a high-performance framework that utilizes a Dynamic Memory Network (DMN) to maintain an object-centric memory bank and an Adaptive Object Alignment (AOA) module to handle pose variations. The proposed method achieves state-of-the-art results on DAVIS 2017 and YouTube-VOS benchmarks while maintaining a fast inference speed of 0.16s per frame.

Problem Statement: The Hurdles of Object-Matching

Traditional object-matching VOS (OVOS) methods face two primary challenges that hinder their accuracy compared to pixel-matching counterparts:

  1. Incomplete Annotations: Initial object masks often only capture a partial view. As an object rotates or changes scale, new parts appear that were not present in the initial template.
  2. Misalignment: Objects are dynamic; changes in pose or camera angle mean that features in the current frame may not spatially align with the stored template, leading to matching failures.
Challenges in VOS: Incomplete annotations and spatial misalignment.

Proposed Methodology

The authors propose a modular framework designed to bridge the gap between efficiency and accuracy. The system processes past frames into a memory bank and uses object proposals in the current frame for matching.

The DMN-AOA framework architecture.

Dynamic Memory Network (DMN)

To address incomplete annotations, the DMN replaces static templates with a dynamic, object-centric memory. It uses a query from the current frame to attend to a bank of past features, allowing the model to "remember" and recognize new parts of an object as they are revealed over time.

Adaptive Object Alignment (AOA)

The AOA module solves the misalignment problem by computing dense correspondences between proposals and the template. It employs a region translation function to spatially "re-pose" features, ensuring that the matching process is robust to non-rigid deformations and pose changes.

Experimental Results

The model, referred to as DMN-AOA, was evaluated on major VOS benchmarks:

  • DAVIS 2017: Achieved a mean $\mathcal{J} \& \mathcal{F}$ score of 84.5% on the validation set, surpassing previous state-of-the-art methods like CFBI and KMN.
  • YouTube-VOS 2018: Recorded an overall score of 82.7%, demonstrating strong generalization capabilities across diverse object categories.
  • Efficiency: By utilizing ROI pooling and a streamlined matching process, the framework maintains a high inference speed of 0.16 seconds per frame, making it significantly faster than many high-accuracy pixel-matching models.

Conclusion

By integrating dynamic memory and adaptive alignment, this work demonstrates that object-matching VOS can achieve competitive accuracy with superior efficiency. Future work may involve tighter integration of proposal generation to further enhance robustness in complex scenes.

Summary updated

Abstract

In this paper, we propose a novel solution for object-matching based semi-supervised video object segmentation, where the target object masks in the first frame are provided. Existing object-matching based methods focus on the matching between the raw object features of the current frame and the first/previous frames. However, two issues are still not solved by these object-matching based methods. As the appearance of the video object changes drastically over time, 1) unseen parts/details of the object present in the current frame, resulting in incomplete annotation in the first annotated frame (e.g. view/scale changes). 2) even for the seen parts/details of the object in the current frame, their positions change relatively (e.g. pose changes/camera motion), leading to a misalignment for the object matching. To obtain the complete information of the target object, we propose a novel object-based dynamic memory network that exploits visual contents of all the past frames. To solve the misalignment problem caused by position changes of visual contents, we propose an adaptive object alignment module by incorporating a region translation function that aligns object proposals towards templates in the feature space. Our method achieves state-of-the-art results on latest benchmark datasets DAVIS 2017 ($\mathcal{J}$ of 81.4% and $\mathcal{F}$ of 87.5% on the validation set) and YouTube-VOS (the overall score of 82.7% on the validation set) with a very efficient inference time (0.16 second/frame on DAVIS 2017 validation set). Code is available at: https://github.com/liang4sx/DMN-AOA.
BibTeX
@inproceedings{liang2021video,
  author = {Shuxian Liang and Xu Shen and Jianqiang Huang and Xian-Sheng Hua},
  title = {{Video Object Segmentation with Dynamic Memory Networks and Adaptive Object Alignment}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00796},
}
Figure