Query-Memory Re-Aggregation for Weakly-supervised Video Object Segmentation

Fanchao Lin, Hongtao Xie, Yan Li, Yongdong ZhangAAAI 2021 · OpenReview

Abstract

Weakly-supervised video object segmentation (WVOS) is an emerging task that tracks and segments a target given only a simple bounding box label. However, existing WVOS methods often fall short in either speed or accuracy because they rely primarily on the exemplar frame, neglecting temporal references from other frames. To address this, we propose a novel Re-Aggregation framework. Our method utilizes feature matching to efficiently locate targets and captures temporal dependencies across multiple frames to guide segmentation. Built on a two-stage structure, the framework establishes an information-symmetric matching process. We introduce a Query-Memory Aggregation (QMA) module in each stage for bidirectional feature aggregation, which mitigates misguidance common in unidirectional approaches. Furthermore, a Cascaded Refinement Module (CRM) combines predictions from different stages to boost performance. Experimental results demonstrate state-of-the-art performance, achieving an overall score of 84.7% on the DAVIS 2016 validation set.

Introduction

Video Object Segmentation (VOS) involves tracking and outlining specific objects throughout a video sequence. Traditionally, this task is approached via semi-supervised or unsupervised learning. Semi-supervised methods require pixel-perfect masks for the initial frame, which are labor-intensive to produce. Conversely, unsupervised methods require no labels but often struggle with complex scenes containing multiple objects.

Comparison of different supervision levels in Video Object Segmentation (VOS).

Weakly-supervised VOS (WVOS) offers a middle ground by requiring only a bounding box for the target object. Despite its potential, current WVOS techniques face a "Goldilocks" problem: they are either fast but inaccurate (relying only on the first frame) or accurate but slow (using complex multi-stage detection pipelines). The primary challenge is information asymmetry: while the "memory" (past frames) contains precise target information, the "query" (current frame) lacks this context, making robust matching difficult in a single pass.

Methodology

The proposed framework addresses information asymmetry through a two-stage Query-Memory Re-Aggregation process.

The proposed two-stage Query-Memory Re-Aggregation framework.

Two-Stage Re-Aggregation

  1. Stage 1 (The Tracker): The model performs an initial asymmetric match between memory frames and the query frame. This stage generates a rough bounding box map ($B_q$), providing the query frame with a "prior" of the target's location.
  2. Stage 2 (The Segmenter): With the target location now known in the query frame, the model performs a symmetric matching process. Using a Siamese-style encoder, both query and memory features are treated identically, leading to a highly precise segmentation mask ($S_q$).

Key Modules

  • Query-Memory Aggregation (QMA): This module performs pixel-wise similarity calculations across time. It includes a Bidirectional Channel Aggregation (BCA) component that adaptively weights feature channels by analyzing both query and memory simultaneously. This helps suppress background noise and focus on target-specific features.
  • Cascaded Refinement Module (CRM): The CRM integrates predictions from both stages. It utilizes CoordConv—adding spatial coordinates to feature maps—to enhance the network's location awareness and refine object boundaries.

To maintain efficiency, query features from the second stage are stored and reused as memory for subsequent frames, eliminating redundant computations.

Evaluation and Results

The model was evaluated on three major benchmarks: DAVIS 2016, DAVIS 2017, and YouTube-VOS.

  • Accuracy: On the DAVIS 2016 validation set, the method achieved a J&F mean score of 84.7%, setting a new state-of-the-art for weakly-supervised methods.
  • Efficiency: The framework operates at 12.1 FPS on a single V100 GPU. This represents a superior balance between speed and accuracy, being nearly 9x faster than high-accuracy competitors like BoltVOS (1.4 FPS).
  • Generalization: Strong performance on "unseen" categories in YouTube-VOS indicates that the bidirectional aggregation helps the model learn general tracking principles rather than just memorizing specific object shapes.

Conclusion

The Re-Aggregation framework effectively bridges the gap between simple bounding box labels and high-fidelity segmentation. By introducing an information-symmetric two-stage process and bidirectional feature selection, the model achieves state-of-the-art accuracy while remaining practical for real-world applications. While the model occasionally struggles with heavily overlapping similar targets, it represents a significant step forward in efficient and accurate weakly-supervised video understanding.

Summary updated

Abstract

Weakly-supervised video object segmentation (WVOS) is an emerging video task that can track and segment the target given a simple bounding box label. However, existing WVOS methods are still unsatisfied in either speed or accuracy, since they only use the exemplar frame to guide the prediction while they neglect the reference from other frames. To solve the problem, we propose a novel Re-Aggregation based framework, which uses feature matching to efficiently find the target and capture the temporal dependencies from multiple frames to guide the segmentation. Based on a two-stage structure, our framework builds an information-symmetric matching process to achieve robust aggregation. In each stage, we design a Query-Memory Aggregation (QMA) module to gather features from the past frames and make bidirectional aggregation to adaptively weight the aggregated features, which relieves the latent misguidance in unidirectional aggregation. To further exploit the information from different aggregation stages, we propose a novel coarse-fine constraint by using the Cascaded Refinement Module (CRM) to combine the predictions from different stages and further boosts the performance. Experimental results on three benchmarks show that our method achieves the state-of-the-art performance in WVOS (e.g., an overall score of 84.7% on the DAVIS 2016 validation set).
BibTeX
@inproceedings{lin2021query,
  author = {Fanchao Lin and Hongtao Xie and Yan Li and Yongdong Zhang},
  title = {{Query-Memory Re-Aggregation for Weakly-supervised Video Object Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=y1LW6kfRDry},
}
Figure