Abstract
Weakly-supervised video object segmentation (WVOS) is an emerging task that tracks and segments a target given only a simple bounding box label. However, existing WVOS methods often fall short in either speed or accuracy because they rely primarily on the exemplar frame, neglecting temporal references from other frames. To address this, we propose a novel Re-Aggregation framework. Our method utilizes feature matching to efficiently locate targets and captures temporal dependencies across multiple frames to guide segmentation. Built on a two-stage structure, the framework establishes an information-symmetric matching process. We introduce a Query-Memory Aggregation (QMA) module in each stage for bidirectional feature aggregation, which mitigates misguidance common in unidirectional approaches. Furthermore, a Cascaded Refinement Module (CRM) combines predictions from different stages to boost performance. Experimental results demonstrate state-of-the-art performance, achieving an overall score of 84.7% on the DAVIS 2016 validation set.
Introduction
Video Object Segmentation (VOS) involves tracking and outlining specific objects throughout a video sequence. Traditionally, this task is approached via semi-supervised or unsupervised learning. Semi-supervised methods require pixel-perfect masks for the initial frame, which are labor-intensive to produce. Conversely, unsupervised methods require no labels but often struggle with complex scenes containing multiple objects.
Weakly-supervised VOS (WVOS) offers a middle ground by requiring only a bounding box for the target object. Despite its potential, current WVOS techniques face a "Goldilocks" problem: they are either fast but inaccurate (relying only on the first frame) or accurate but slow (using complex multi-stage detection pipelines). The primary challenge is information asymmetry: while the "memory" (past frames) contains precise target information, the "query" (current frame) lacks this context, making robust matching difficult in a single pass.
Methodology
The proposed framework addresses information asymmetry through a two-stage Query-Memory Re-Aggregation process.
Two-Stage Re-Aggregation
- Stage 1 (The Tracker): The model performs an initial asymmetric match between memory frames and the query frame. This stage generates a rough bounding box map ($B_q$), providing the query frame with a "prior" of the target's location.
- Stage 2 (The Segmenter): With the target location now known in the query frame, the model performs a symmetric matching process. Using a Siamese-style encoder, both query and memory features are treated identically, leading to a highly precise segmentation mask ($S_q$).
Key Modules
- Query-Memory Aggregation (QMA): This module performs pixel-wise similarity calculations across time. It includes a Bidirectional Channel Aggregation (BCA) component that adaptively weights feature channels by analyzing both query and memory simultaneously. This helps suppress background noise and focus on target-specific features.
- Cascaded Refinement Module (CRM): The CRM integrates predictions from both stages. It utilizes CoordConv—adding spatial coordinates to feature maps—to enhance the network's location awareness and refine object boundaries.
To maintain efficiency, query features from the second stage are stored and reused as memory for subsequent frames, eliminating redundant computations.
Evaluation and Results
The model was evaluated on three major benchmarks: DAVIS 2016, DAVIS 2017, and YouTube-VOS.
- Accuracy: On the DAVIS 2016 validation set, the method achieved a J&F mean score of 84.7%, setting a new state-of-the-art for weakly-supervised methods.
- Efficiency: The framework operates at 12.1 FPS on a single V100 GPU. This represents a superior balance between speed and accuracy, being nearly 9x faster than high-accuracy competitors like BoltVOS (1.4 FPS).
- Generalization: Strong performance on "unseen" categories in YouTube-VOS indicates that the bidirectional aggregation helps the model learn general tracking principles rather than just memorizing specific object shapes.
Conclusion
The Re-Aggregation framework effectively bridges the gap between simple bounding box labels and high-fidelity segmentation. By introducing an information-symmetric two-stage process and bidirectional feature selection, the model achieves state-of-the-art accuracy while remaining practical for real-world applications. While the model occasionally struggles with heavily overlapping similar targets, it represents a significant step forward in efficient and accurate weakly-supervised video understanding.
Summary updated