Abstract
Video Object Segmentation (VOS) involves tracking and segmenting specific objects across video sequences. Current approaches are often divided into pixel-matching and object-matching methods. While object-matching is efficient, it historically struggled with evolving object appearances and spatial misalignments. This paper introduces a high-performance framework that utilizes a Dynamic Memory Network (DMN) to maintain an object-centric memory bank and an Adaptive Object Alignment (AOA) module to handle pose variations. The proposed method achieves state-of-the-art results on DAVIS 2017 and YouTube-VOS benchmarks while maintaining a fast inference speed of 0.16s per frame.
Problem Statement: The Hurdles of Object-Matching
Traditional object-matching VOS (OVOS) methods face two primary challenges that hinder their accuracy compared to pixel-matching counterparts:
- Incomplete Annotations: Initial object masks often only capture a partial view. As an object rotates or changes scale, new parts appear that were not present in the initial template.
- Misalignment: Objects are dynamic; changes in pose or camera angle mean that features in the current frame may not spatially align with the stored template, leading to matching failures.
Proposed Methodology
The authors propose a modular framework designed to bridge the gap between efficiency and accuracy. The system processes past frames into a memory bank and uses object proposals in the current frame for matching.
Dynamic Memory Network (DMN)
To address incomplete annotations, the DMN replaces static templates with a dynamic, object-centric memory. It uses a query from the current frame to attend to a bank of past features, allowing the model to "remember" and recognize new parts of an object as they are revealed over time.
Adaptive Object Alignment (AOA)
The AOA module solves the misalignment problem by computing dense correspondences between proposals and the template. It employs a region translation function to spatially "re-pose" features, ensuring that the matching process is robust to non-rigid deformations and pose changes.
Experimental Results
The model, referred to as DMN-AOA, was evaluated on major VOS benchmarks:
- DAVIS 2017: Achieved a mean $\mathcal{J} \& \mathcal{F}$ score of 84.5% on the validation set, surpassing previous state-of-the-art methods like CFBI and KMN.
- YouTube-VOS 2018: Recorded an overall score of 82.7%, demonstrating strong generalization capabilities across diverse object categories.
- Efficiency: By utilizing ROI pooling and a streamlined matching process, the framework maintains a high inference speed of 0.16 seconds per frame, making it significantly faster than many high-accuracy pixel-matching models.
Conclusion
By integrating dynamic memory and adaptive alignment, this work demonstrates that object-matching VOS can achieve competitive accuracy with superior efficiency. Future work may involve tighter integration of proposal generation to further enhance robustness in complex scenes.
Summary updated